Pith. sign in

REVIEW 4 major objections 5 minor 82 references

LMMs can generalize to thermal, depth, X-ray and other unseen visual modalities without any real non-RGB training data by learning from fabricated synthetic images plus textual modality contexts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:44 UTC pith:TFGUOKUI

load-bearing objection Solid empirical recipe for zero-shot non-RGB LMM gains via RGB fabrication + modality contexts; real-modality transfer is real but rests on a simple proxy and choice-VQA. the 4 major comments →

arxiv 2607.10308 v1 pith:TFGUOKUI submitted 2026-07-11 cs.CV

Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

Shihao Yuan , Yuanze Li , Ruyi Zhang , Ming Liu , Wangmeng Zuo This is my paper
classification cs.CV
keywords large multimodal modelsvisual modality generalizationsynthetic datainstruction tuningmodality synthesiszero-shot adaptationVVM-TuningVVM-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current large multimodal models excel at RGB images but struggle when the input is thermal, depth, X-ray or any other visual modality they never saw in training. The authors argue that every visual modality is simply a different sampling of the same physical scene, so a model only needs two skills: the ability to read invariant scene semantics no matter how the image looks, and the ability to map those looks back to modality-specific physical meanings when given a short textual description of the modality. They show that both skills can be taught entirely from RGB images by fabricating appearance-varied synthetic images and pairing them with invented modality contexts, then instruction-tuning the model on disentangled perception and understanding tasks. Five different base models, after this synthetic-only fine-tuning, improve on both real non-RGB modalities and completely novel synthetic modalities, averaging roughly eight points on perception and four points on understanding. The result matters because it suggests that expensive paired non-RGB datasets are not a prerequisite for expanding the visual reach of multimodal models.

Core claim

By training solely on images synthesized from ordinary RGB scenes (via histogram remapping, scientific colormaps and noise) together with fabricated textual modality contexts, large multimodal models acquire both modality-unaware semantic perception and context-guided modality understanding, yielding consistent zero-shot gains on real thermal, depth, X-ray and optical-flow images as well as on novel synthetic modalities, without ever seeing real non-RGB training data.

What carries the argument

VVM-Tuning: a two-stage instruction-tuning regime that first uses fabricated appearance-varied images to disentangle invariant scene semantics from photographic elements, then injects synthetic modality contexts so the model learns to re-map those elements onto modality-specific physical attributes at inference time.

Load-bearing premise

The simple image-fabrication pipeline plus hand- or model-written modality contexts is a faithful enough proxy for real sensor physics that skills learned on the fabrications transfer to true thermal, depth and X-ray images.

What would settle it

Fine-tune the same base models on an equal amount of real thermal/depth/X-ray data and measure whether the synthetic-only models still match or exceed that real-data baseline on held-out real modalities; a large gap would falsify the claim that fabrication is sufficient.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can extend existing RGB-centric LMMs to new visual sensors by writing a short modality context and running a cheap synthetic fine-tune, without collecting paired non-RGB data.
  • Future LMM training recipes can deliberately separate semantic alignment from appearance alignment, reducing over-fitting to RGB statistics.
  • Benchmarks for multimodal models should include both real and fabricated non-RGB modalities if they claim general visual competence.
  • Modality contexts become a lightweight inference-time interface for adapting a single model to many imaging devices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If fabrication quality is the bottleneck, more physically accurate sensor simulators (rather than more RGB data) may be the next leverage point for further gains.
  • The same disentanglement idea could apply to non-visual modalities such as audio spectrograms or point-cloud projections that also share underlying scene structure.
  • Once models reliably use modality contexts, interactive systems could let users invent ad-hoc imaging meanings on the fly and obtain immediate zero-shot interpretations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that LMMs can generalize to unseen visual modalities without in-modality training by learning modality-unaware perception (disentangled semantic vs. photographic alignments) and modality-aware understanding via synthetic data. VVM-Tuning fabricates appearance-varied images from RGB (grayscale + random cubic-spline histogram remapping + scientific colormaps + augmentations) and synthesizes textual modality contexts; models are instruction-tuned on description/VQA tasks that separate invariant semantics from appearance and then map appearances to physical attributes under context. A new VVM-Bench (6 modalities: thermal/depth/X-ray from VS-TDX, optical-flow visualizations, and two fully synthetic styles; 11 VQA tasks) evaluates perception (no context) and understanding (with context). Across five base models (LLaVA-1.5, Qwen2.5-VL 3B/7B, Qwen3-VL 4B/8B), synthetic-only SFT yields average gains of +8.2% on perception and +3.8% on understanding, including on real non-RGB modalities, with ablations on task components, context presence, data scale, and small real-data fine-tuning.

Significance. If the transfer claim holds, the work opens a practical route to pan-modal LMM capability without costly paired non-RGB corpora, and the disentanglement of semantic vs. photographic perception is a useful conceptual contribution. Strengths include multi-model consistency (Tables 1–2), clear ablations separating perception vs. understanding tasks (Table 4) and with/without modality context (Table 3), supplementary data-scale and small real-TDX comparisons, qualitative examples, and the promise of public code and data. VVM-Bench fills a genuine evaluation gap relative to RGB-centric suites and specialized domain benchmarks. The result is outside current practice (most non-RGB LMM work is in-modality fine-tuning) but is not internally inconsistent; the main risk is over-claiming pure zero-shot transfer from a simple fabrication proxy.

major comments (4)
  1. [§3.2, Fig. 3; §4.3; Tables 1–2] The load-bearing zero-shot claim on real thermal/depth/X-ray rests on the §3.2 fabrication pipeline (grayscale + random cubic-spline histogram + 22 colormaps + noise/blur) plus synthetic contexts being a faithful enough proxy for real sensor physics. §4.3 already notes a “synthetic gap” (Optical Flow improves most; fully synthetic modalities improve least on understanding). The manuscript needs a stronger isolation experiment: e.g., controlled comparison of transfer when training colormaps/histograms are deliberately mismatched to evaluation sensors, or feature-level analysis showing what is learned beyond residual RGB knowledge already present in modern bases. Without this, the real-modality gains remain consistent with improved exploitation of pre-existing non-RGB exposure rather than pure synthetic transfer.
  2. [Table 3; §4.4] Table 3 shows that simply inserting human-written modality context produces large baseline jumps (up to +39.6% on Depth; average Δs of +15.4% Thermal, +32.8% Depth, +3.1% X-ray). After VVM-Tuning the “w/o M.C.” columns also rise, which is consistent with better perception but also with better use of residual knowledge of common non-RGB modalities that larger/more recent models almost certainly saw during pre-training/SFT. The paper should quantify residual knowledge more carefully (e.g., probe base models for thermal/X-ray recognition before any tuning; report results on a base known to be RGB-only if available) and clarify how much of the claimed zero-shot gain is attributable to the synthetic training versus context prompting alone.
  3. [§3.5; §4.1; Appendix A.5 / B] VVM-Bench evaluation is exclusively multiple-choice VQA with forced letter output. The limitations appendix correctly notes that this format is “easy to be hacked by guessing” on understanding tasks, especially when rich modality contexts are supplied. Given that understanding gains are already modest (+3.8% average) and that Optical Flow (a visualization the pipeline can closely mimic) improves most, the central generalization claim would be more convincing with at least one open-ended or multi-hop metric, or with an explicit analysis of option-bias / random-guess baselines per task. Choice accuracy alone is insufficient to fully support the modality-aware understanding claim.
  4. [§3.2–3.5; Tables 1–2; Appendix C] Training and evaluation both rely on similarly fabricated images and LMM-generated contexts for Synthesized 1/2 (and partly for Optical Flow). While the real VS-TDX modalities are external and therefore the primary evidence, the paper still reports and averages synthetic-test numbers that risk mild self-consistency inflation. Either de-emphasize Synthesized 1/2 in the main averages or add a held-out fabrication style (different colormap family, different histogram process, or a third-party style-transfer method never seen in training) as a true out-of-distribution synthetic test.
minor comments (5)
  1. [§1, §3.3–3.4, Tables 1–4] Notation and terminology drift slightly: “modality-unaware perception,” “photographic perception,” “semantic perception,” and “modality-aware understanding” are introduced carefully in §1–3 but later abbreviated inconsistently in tables and captions; a short glossary or consistent abbreviations would help.
  2. [§3.2; Appendix C.1] Figure 3 and Algorithm A1 describe the spline histogram procedure, but the main text does not state the exact hyper-parameters used for the training set versus the “changed hyper-parameters” for Synthesized 2; a one-sentence clarification would improve reproducibility.
  3. [§2] Related-work coverage of concurrent non-RGB LMM efforts (Infrared-LLaVA, SpatialBot, EarthGPT, etc.) is adequate but could more sharply distinguish “in-modality specialization” from the paper’s zero-shot claim; a short comparison table of training data requirements would strengthen the positioning.
  4. [Throughout; Appendix] Typos and minor English issues appear (e.g., “beseene,” “reinterprete,” “tow description,” “Illuminance Llevel” in the appendix); a careful proofread is needed before camera-ready.
  5. [§4.1] The 50k training mixture retains 12k RGB instruction data; the paper should state the source of those RGB samples and confirm they do not overlap with VVM-Bench images.

Circularity Check

1 steps flagged

Mild self-consistency risk only on the synthetic half of VVM-Bench (same/similar fabrication pipeline as training); load-bearing real-modality gains use external VS-TDX/RAFT data and are not forced by construction.

specific steps
  1. other [§3.5 Benchmark + §4.2–4.3 (Tables 1–2)]
    "The other two visual modalities are completely synthesized as unseen visual modalities. The images in Synthesized 1 are fabricated using a style transfer algorithm, StyleShot [11] … As for images for Synthesized 2, we use the same image synthesis method as described in Sec. 3.2 with changed hyper-parameters … Experiments demonstrate that, via our training on synthetic modalities, 5 tested models exhibit consistent improvements on both real-world and novel synthetic modalities"

    Training data are produced by the Sec. 3.2 pipeline (grayscale + random spline histogram + 22 colormaps + noise/blur). Synthesized 2 re-uses that exact pipeline (only hyper-parameters changed) and Synthesized 1 uses a closely related appearance-alteration method. Accuracy gains reported on those two subsets are therefore partly expected by construction (in-distribution for the fabrication process) rather than independent evidence of generalization to a wholly novel imaging process. This does not affect the real-modality (VS-TDX/RAFT) numbers that carry the main claim.

full rationale

This is an empirical instruction-tuning paper, not a first-principles derivation. There are no equations, fitted parameters renamed as predictions, uniqueness theorems, or load-bearing self-citations that reduce the central claim to its inputs. Training fabricates images via grayscale + random cubic-spline histogram + colormaps (Sec. 3.2) and synthesizes modality contexts; evaluation reports accuracy lifts on both real modalities (thermal/depth/X-ray from VS-TDX, optical flow from RAFT) and two fabricated modalities. The real-modality numbers and the w/o-modality-context ablations (Table 3) are independent of the training synthesis and therefore not circular. The only mild issue is that Synthesized 2 re-uses the identical image-synthesis pipeline (with only hyper-parameter changes) and Synthesized 1 uses a related style-transfer method, so gains on those two subsets are partly in-distribution for the fabrication process rather than pure zero-shot to a wholly novel sensor. That does not force the headline claim (zero-shot gains on real modalities without in-modality data) and does not match any of the high-priority circularity patterns. Score 2 reflects this limited self-consistency on the synthetic evaluation slice only.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

The central claim rests on a domain story (modalities as samplings of one world), a simple hand-designed image synthesis process with several free design choices, and the premise that short textual modality contexts plus instruction tuning suffice for zero-shot sensor adaptation. No new physical entity is postulated; the invented pieces are methodological (pipeline, task split, bench). Free parameters are training and synthesis knobs, not fitted physical constants. The ledger is modest for an empirical ML paper but the transfer assumption is load-bearing.

free parameters (5)
  • SFT learning rate = 1e-5
    Set to 1e-5 by authors; not derived; affects whether gains appear.
  • Training mixture size and RGB retention = ~50k total / 12k RGB / 1 epoch
    ~50k total samples with 12k retained RGB instructions for 1 epoch; composition is a design choice that preserves general ability and drives reported deltas.
  • Histogram spline keypoints and rejection sampling = 1–4 keypoints (N_max design choice)
    N randomly in 1..N_max with rejection until map stays in [0,255]; controls appearance diversity of fabricated images.
  • Color map set and augmentations = 22 colormaps + noise/blur/smooth
    22 scientific colormaps plus noise/smooth/blur chosen by hand to mimic non-RGB look; defines the synthetic training distribution.
  • Batch size / hardware schedule = 32
    Total batch 32 on 4×A800; standard but free relative to the claim.
axioms (5)
  • domain assumption Different visual modalities are distinct samplings of the same physical world and therefore share modality-agnostic scene semantics.
    Stated in Abstract and §1 as the foundation for pan-modal generalization; not proved, motivates the whole method.
  • domain assumption Basic visual elements (color, edge, contour, shape, position) remain perceivable across 3-channel non-RGB images without modality knowledge (modality-unaware / photographic perception).
    §1 and Fig. 2 human-vision analogy; justifies disentangled semantic vs photographic tasks.
  • ad hoc to paper Textual modality contexts provided at inference, after instruction tuning on synthetic contexts, are sufficient for zero-shot mapping of appearance to modality-specific physical attributes.
    Core of modality-aware understanding (§3.4); success is empirical, not derived from a theory of in-context sensor physics.
  • ad hoc to paper Simple RGB→grayscale→spline histogram→colormap fabrication produces training signals that transfer to real thermal/depth/X-ray sensors.
    §3.2 pipeline; authors later note a residual synthetic gap for meanings vs appearances (§4.3).
  • domain assumption Standard supervised fine-tuning cross-entropy on instruction data (Eq. 1) is an adequate objective for installing both perception and context-use skills.
    §3.1; conventional LMM SFT assumption.
invented entities (4)
  • VVM-Tuning framework no independent evidence
    purpose: Two-part instruction-tuning recipe (modality-unaware perception + modality-aware understanding) for zero-shot non-RGB generalization.
    Methodological construct; evaluated only via the paper’s own bench and baselines.
  • Fabricated modality synthesis pipeline no independent evidence
    purpose: Generate diverse non-RGB-like images and synthetic modality contexts from RGB sources.
    Hand-designed process (Fig. 3, Alg. A1); no external sensor model validates fidelity beyond empirical transfer.
  • VVM-Bench no independent evidence
    purpose: 6-modality, 11-task VQA benchmark for perception and understanding under optional modality contexts.
    New evaluation suite assembled partly from VS-TDX and partly fabricated; primary evidence for the claim is defined on this bench.
  • Modality context (as LMM prompt object for unseen sensors) no independent evidence
    purpose: Textual description of color–physics mapping inserted at inference to enable modality-aware understanding.
    Operationalized in this paper for zero-shot sensor adaptation; related to ordinary prompting but specialized here.

pith-pipeline@v1.1.0-grok45 · 30164 in / 4186 out tokens · 45451 ms · 2026-07-14T12:44:16.665627+00:00 · methodology

0 comments
read the original abstract

Despite the advancements of Large Multimodal Models (LMMs) in RGB vision, their ability to generalize to unseen visual modalities remains a largely unexplored challenge. We argue that different visual modalities are merely distinct samplings of the same physical world. Therefore, effective generalization requires models to possess both modality-agnostic perception of scene semantics and the adaptability to modality-specific characteristics. To achieve this, we propose a training framework, VVM-Tuning, to equip LMMs with these capabilities through modality synthesis and modality contexts. Specifically, we synthesize diverse appearance-varied images from RGB scenes, training the model to disentangle invariant semantics from varying visual appearances, and align these appearances with language for visual concepts decoupled from modalities. We then introduce modality contexts in the prompt and use instruction tuning to assist the model in mapping these appearance variations back to modality-related attributes, enabling zero-shot adaptation to unseen modalities during inference. To facilitate research in this direction, we introduce VVM-Bench, a comprehensive benchmark featuring 6 real and synthetic modalities to evaluate semantic perception and modality understanding. Experiments demonstrate that, via our training on synthetic modalities, 5 tested models exhibit consistent improvements on both real-world and novel synthetic modalities without in-modality training. Source code and data will be publicly available at https://github.com/Hunter-Will/VVM-Tuning.

Figures

Figures reproduced from arXiv: 2607.10308 by Ming Liu, Ruyi Zhang, Shihao Yuan, Wangmeng Zuo, Yuanze Li.

Figure 1
Figure 1. Figure 1: An overview of our idea. Different visual modalities (Thermal and RGB) are both signal samplings of our physical world, sharing the same underlying seman￾tics (formed by the person and the car). They also share basic visual concepts, as the color itself is invariant across modalities. The modality-specific physical meaning is encoded by the rearrangement and remapping of basic visual concepts. Thus, we int… view at source ↗
Figure 2
Figure 2. Figure 2: An example of how human vision adapts to unseen visual modalities. Our eyes can capture basic visual elements regardless of modalities, and our brain recognizes the semantics even through a few visual elements. This demonstrates a compatible modality-unaware perception, which we could implement in LMMs’ training. generalized among diverse visual modalities, as long as they are 3-channel digital images. Tha… view at source ↗
Figure 3
Figure 3. Figure 3: An overview of our modality synthesis pipeline, composed of the image synthesis and modality context fabrication. 3 Methods 3.1 Preliminary From the base idea of how different visual modalities follow the basic imag￾ing principles to map the real-world scenarios into photos, our method tries to imitate the same process of how humans understand versatile visual modalities, which leads to two different abili… view at source ↗
Figure 4
Figure 4. Figure 4: Different examples of fabricated images with different color maps produced by our image synthesis pipeline [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: The construction of modality description is achieved by leveraging a large [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An overview of VVM-Bench. Our VVM-Bench contains 6 visual modalities and 11 VQA tasks for perception and understanding evaluation, including 3 real modalities collected from previous work and 3 fabricated from RGB images. RGB images, including Optical Flow, Synthesized 1, and Synthesized 2. Specif￾ically, the optical flow images are visualizations of predicted optical flows from RAFT [31] to serve as signa… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

82 extracted references · 12 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2509.23661 (2025)

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al.: LLaVA-OneVision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    In: Proceedings of the IEEE International Con- ference on Computer Vision

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: Proceedings of the IEEE International Con- ference on Computer Vision. pp. 2425–2433 (2015)

    2015

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [5]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Cai, W., Ponomarenko, I., Yuan, J., Li, X., Yang, W., Dong, H., Zhao, B.: Spatial- Bot: Precise Spatial Understanding with Vision Language Models. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 9490–9498. IEEE (2025)

    2025

  5. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu,X.,Lu,L.,etal.:InternVL:ScalingupVisionFoundationModelsandAligning for Generic Visual-Linguistic Tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24185–24198 (2024)

    2024

  6. [7]

    arXiv preprint arXiv:2412.20750 (2024)

    Chung, S., Yu, Y., Kim, S.Y., Chee, Y., Ro, Y.M.: Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Bench- marking. arXiv preprint arXiv:2412.20750 (2024)

    Pith/arXiv arXiv 2024

  7. [8]

    arXiv preprint arXiv:2502.03333 (2025)

    Deperrois, N., Matsuo, H., Ruipérez-Campillo, S., Vandenhirtz, M., Laguna, S., Ryser, A., Fujimoto, K., Nishio, M., Sutter, T.M., Vogt, J.E., et al.: RadVLM: A Multitask Conversational Vision-Language Model for Radiology. arXiv preprint arXiv:2502.03333 (2025)

    arXiv 2025

  8. [9]

    arXiv preprint arXiv:2306.13394 (2023)

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023)

    Pith/arXiv arXiv 2023

  9. [10]

    arXiv preprint arXiv:2411.15296 (2024)

    Fu, C., Zhang, Y.F., Yin, S., Li, B., Fang, X., Zhao, S., Duan, H., Sun, X., Liu, Z., Wang, L., et al.: MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs. arXiv preprint arXiv:2411.15296 (2024)

    Pith/arXiv arXiv 2024

  10. [11]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 16 Shihao Yuan, Yuanze Li,et al

    Gao, J., Sun, Y., Liu, Y., Tang, Y., Zeng, Y., Qi, D., Chen, K., Zhao, C.: StyleShot: A Snapshot on Any Style. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 16 Shihao Yuan, Yuanze Li,et al

    2025

  11. [12]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: ImageBind: One Embedding Space To Bind Them All. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15180– 15190 (2023)

    2023

  12. [13]

    In: 6th International Symposium on Advanced Optical Manufacturing and Testing Technologies: Large Mirrors and Telescopes

    Hu, J., Peng, X., Xu, Z.: Study of Gray Image Pseudo-Color Processing algorithms. In: 6th International Symposium on Advanced Optical Manufacturing and Testing Technologies: Large Mirrors and Telescopes. vol. 8415, pp. 323–328. SPIE (2012)

    2012

  13. [14]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024

    Jiang, S., Chen, Z., Liang, J., Zhao, Y., Liu, M., Qin, B.: Infrared-LLaVA: En- hancing Understanding of Infrared Images in Multi-Modal Large Language Mod- els. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 8573–8591 (2024)

    2024

  14. [15]

    IEEE Transactions on Pattern Analysis and Machine Intelligence35(12), 2891– 2903 (2013)

    Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: BabyTalk: Understanding and Generating Simple Image Descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence35(12), 2891– 2903 (2013)

    2013

  15. [16]

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., Li, C.: LLaVA-OneVision: Easy Visual Task Transfer (2024)

    2024

  16. [17]

    arXiv preprint arXiv:2511.21631 (2025)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Qwen2.5-VL Technical Report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  17. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: SEED- Bench: Benchmarking Multimodal Large Language Models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13299– 13308 (2024)

    2024

  18. [19]

    Advances in neural information processing systems36(2024)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual Instruction Tuning. Advances in neural information processing systems36(2024)

    2024

  19. [20]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: MMBench: Is Your Multi-modal Model an All-around Player? In: European Conference on Computer Vision. pp. 216–233. Springer (2024)

    2024

  20. [21]

    IEEE Transactions on Geoscience and Remote Sensing 58(12), 8555–8566 (2020)

    Lobry, S., Marcos, D., Murray, J., Tuia, D.: RSVQA: Visual Question Answering for Remote Sensing Data. IEEE Transactions on Geoscience and Remote Sensing 58(12), 8555–8566 (2020)

    2020

  21. [22]

    In: Interna- tional Conference on Learning Representations (2017)

    Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. In: Interna- tional Conference on Learning Representations (2017)

    2017

  22. [23]

    In: The 36th Conference on Neural Information Pro- cessing Systems (NeurIPS) (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In: The 36th Conference on Neural Information Pro- cessing Systems (NeurIPS) (2022)

    2022

  23. [24]

    In: Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision

    Mathew, M., Karatzas, D., Jawahar, C.: DocVQA: A Dataset for VQA on Docu- ment Images. In: Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision. pp. 2200–2209 (2021)

    2021

  24. [25]

    College of the Redwoods 45(1), 1049–1060 (1998)

    McKinley, S., Levine, M.: Cubic Spline Interpolation. College of the Redwoods 45(1), 1049–1060 (1998)

    1998

  25. [26]

    arXiv preprint arXiv:2503.19654 (2025)

    Moshtaghi, M., Khajavi, S.H., Pajarinen, J.: RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Models. arXiv preprint arXiv:2503.19654 (2025)

    Pith/arXiv arXiv 2025

  26. [27]

    arXiv preprint arXiv:2304.03277 (2023)

    Peng, B., Li, C., He, P., Galley, M., Gao, J.: Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277 (2023)

    Pith/arXiv arXiv 2023

  27. [28]

    In: Acquisition and Analysis of Pictorial Data

    Radewan, C.H.: Digital Image Processing With Pseudo-Color. In: Acquisition and Analysis of Pictorial Data. vol. 48, pp. 50–56. SPIE (1975) Generalize LMMs to Versatile Visual Modalities 17

    1975

  28. [29]

    In: Spline Functions and Approximation Theory: Proceedings of the Symposium held at the University of Alberta, Edmon- ton May 29 to June 1, 1972

    Schoenberg, I.: Splines and Histograms. In: Spline Functions and Approximation Theory: Proceedings of the Symposium held at the University of Alberta, Edmon- ton May 29 to June 1, 1972. pp. 277–327. Springer (1973)

    1972

  29. [30]

    In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp

    Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., Cai, D.: PandaGPT: One Model To Instruction-Follow Them All. In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp. 11–23 (2023)

    2023

  30. [31]

    In: European Conference on Computer Vision

    Teed, Z., Deng, J.: RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In: European Conference on Computer Vision. pp. 402–419. Springer (2020)

    2020

  31. [32]

    In: Proceedings of the 23rd Work- shop on Biomedical Natural Language Processing

    Thawakar, O.C., Shaker, A.M., Mullappilly, S.S., Cholakkal, H., Anwer, R.M., Khan, S., Laaksonen, J., Khan, F.: XrayGPT: Chest Radiographs Summarization using Large Medical Vision-Language Models. In: Proceedings of the 23rd Work- shop on Biomedical Natural Language Processing. pp. 440–448 (2024)

    2024

  32. [33]

    arXiv preprint arXiv:2508.18265 (2025)

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv preprint arXiv:2508.18265 (2025)

    Pith/arXiv arXiv 2025

  33. [34]

    Pattern Recognition137, 109347 (2023)

    Xu, M., Yoon, S., Fuentes, A., Park, D.S.: A Comprehensive Survey of Image Augmentation Techniques for Deep Learning. Pattern Recognition137, 109347 (2023)

    2023

  34. [35]

    arXiv preprint arXiv:2308.02490 (2023)

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv preprint arXiv:2308.02490 (2023)

    Pith/arXiv arXiv 2023

  35. [36]

    arXiv preprint arXiv:2505.11852 (2025)

    Yue, J., Zhang, S., Jia, Z., Xu, H., Han, Z., Liu, X., Wang, G.: MedSG- Bench: A Benchmark for Medical Image Sequences Grounding. arXiv preprint arXiv:2505.11852 (2025)

    Pith/arXiv arXiv 2025

  36. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: MMMU: A Massive Multi-discipline Multimodal Un- derstanding and Reasoning Benchmark for Expert AGI. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9556– 9567 (2024)

    2024

  37. [38]

    IEEE Transactions on Geoscience and Remote Sensing62, 1–20 (2024)

    Zhang, W., Cai, M., Zhang, T., Zhuang, Y., Mao, X.: EarthGPT: A Universal Mul- timodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain. IEEE Transactions on Geoscience and Remote Sensing62, 1–20 (2024)

    2024

  38. [39]

    In: Infrared Materials, Devices, and Applications

    Zhang, X., Bai, T., Li, H.: Pseudo-color coding method of infrared images based on human vision system. In: Infrared Materials, Devices, and Applications. vol. 6835, pp. 403–410. SPIE (2008)

    2008

  39. [40]

    IEEE Transac- tions on Visualization and Computer Graphics22(8), 2051–2069 (2015)

    Zhou, L., Hansen, C.D.: A Survey of Colormaps in Visualization. IEEE Transac- tions on Visualization and Computer Graphics22(8), 2051–2069 (2015)

    2051

  40. [41]

    Perception Mean

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.10479 (2025) Supplementary Material A Additional Results A.1 Quantity Results on VS-TDX As discussed in the main paper, we have collect...

    Pith/arXiv arXiv 2025

  41. [42]

    Make sure you understand the scene fully

    Analyze the two images, ignore the differences between them, focus on the same scene behind them. Make sure you understand the scene fully

  42. [43]

    You can check reference description if you are not sure about the scene, but DO NOT copy it

    Generate a description that covers the scene. You can check reference description if you are not sure about the scene, but DO NOT copy it

  43. [44]

    Note that you are describing the scene not the two images, so do not describe any features that affected by the camera or photographic methods

  44. [45]

    Generate a diverse question that require a subject to describe the scene from a image, please note the subject only can see one image

  45. [46]

    [REQUIREMENT]

    Output the scene description and the question in format of JSON. [REQUIREMENT]

  46. [47]

    Please ensure that the description is consistent with both images and DO NOT add any features that not exists in the images to the description

  47. [48]

    Also DO NOT describe the features that not showing in both images

    The description should focus on the scene in the images, avoid describing details like colors and texture. Also DO NOT describe the features that not showing in both images

  48. [49]

    <caption>

    Give your ouput in a format of JSON, which has the same structure like this:... <image>: RGB image and Fabricated image <caption>: RGB image description 16 Shihao Yuan, Yuanze Li,et al. T able A16:Photographic Description Generation Prompt Template Photographic Description Generation Prompt T emplate <image> You are an expert at understanding image and de...

  49. [50]

    Analyze the image and scene description, make sure you understand the image fully

  50. [51]

    Such features may includes colors, shapes, photographic effects or other features that not involved in the scene description

    Check all elements in the image and write down the features that missing from the scene description. Such features may includes colors, shapes, photographic effects or other features that not involved in the scene description

  51. [52]

    Based on the missing features, generate a description that includes all missing features but does NOT include described features in the scene description

  52. [53]

    Please note the subject only can see the image

    Generate a diverse question that require a subject to describe the image from the colors, shapes and attributes, etc. Please note the subject only can see the image

  53. [54]

    [REQUIREMENT]

    Output the missing features, your description and the question in format of JSON. [REQUIREMENT]

  54. [55]

    Please ensure that your description is consistent with the image and DO NOT add any features that not exists in the image to your description

  55. [56]

    Also DO NOT describe the features that already included in the scene description

    Your description should focus on the missing features in the scene description, especially low-level features like colors and shapes, avoid describing abstract concepts like event and relationship. Also DO NOT describe the features that already included in the scene description

  56. [57]

    Scene description

    Give your ouput in a format of JSON, which has the same structure like this:... <image>: Fabricated image. <caption>: Semantic Description generated before. D.2 Modality-unaware VQA Tasks The categories of modality-unaware VQA tasks are listed in Tab. A17 and Tab. A18 with exemplar questions. As described before, the semantic QA tasks in Tab. A17 are focu...

  57. [59]

    Make a strategy to create challenging questions about the image to meet the requirements as below to test the ability of the subject (a visual assistant)

  58. [60]

    Your question should be clear and meaningful, requiring the subject having a strong ability that connecting common knowledge to the scene to solve the question

    Create one question of the given question type based on the strategy and image. Your question should be clear and meaningful, requiring the subject having a strong ability that connecting common knowledge to the scene to solve the question

  59. [61]

    The answer should be clear and distinct

    Provide the correct answer to your question according to the image. The answer should be clear and distinct

  60. [62]

    Adjust the number of incorrect answers according to your question (i.e

    Generate at most {negative_samples_num} different plausible incorrect answers that the subject might give if it is doesn't understand the scene or lacks abilities. Adjust the number of incorrect answers according to your question (i.e. if your question only has'yes'or'no'as answers, only generate one incorrect answer). [REQUIREMENT]

  61. [63]

    Please ensure that the image information is fully utilized, and the answer cannot be inferred from the question alone

  62. [64]

    Supplementary Material 19

    Make the length of the incorrect answers similar to the correct answer to prevent that the correct answer is always the longest one. Supplementary Material 19

  63. [65]

    You need to generated all kinds of questions related to the question type and the ability behind the question type

    Create questions according to the question type, but please do not completely copy the content of the example questions. You need to generated all kinds of questions related to the question type and the ability behind the question type

  64. [66]

    Avoid generating question and answers with a lot of guessing and assuming

    You can generate diverse questions requiring broad common knowledge, but ensure the connection between question and answer to be clear and reasonable. Avoid generating question and answers with a lot of guessing and assuming

  65. [67]

    strategy

    Give your ouput in a format of JSON, which has the same structure like this: [FORMAT] Your output MUST be in JSON format as follows: { "strategy": "[HOW TO MAKE IT CHALLENGING]", "question": "[YOUR QUESTION]", "answer": "[YOUR CORRECT ANSWER]", "ability_needed": "[ABILITY TESTED OF YOUR QUESTION TYPE]", "incorrect_answers": [ "[INCORRECT ANSWER 1]", "[INC...

  66. [68]

    Analyze the tow descriptions, make sure you understand the image according to the description

  67. [69]

    Analyze the modality description which explained what the color filter is trying to emphasize

  68. [70]

    According to the modality description, combine the scene description and photographic description to a more completed description that cover all features in the image combined with the modality description

  69. [71]

    You can assume the subject only can see the image and modality description

    Create one question that asking a subject describe the image based on the modality description and image. You can assume the subject only can see the image and modality description. [REQUIREMENT]1. Please ensure that the image information and the modality description are fully utilized in the completed description. Every details in the image is described ...

  70. [72]

    However you can make up some extra knowledge in the modality description, it will be provided to subject

    Avoid generating question with a lot of guessing and assuming. However you can make up some extra knowledge in the modality description, it will be provided to subject

  71. [73]

    <photographic_caption>: Photographic Description generated before

    Give your ouput in a format of JSON, which has the same structure like this: <semantic_caption>: Semantic Description generated before. <photographic_caption>: Photographic Description generated before. <modality_context>: Modality Context fabricated before. Supplementary Material 21 T able A21:Modality QA Tasks and Question examples Task Question Example...

  72. [74]

    Analyze the image, scene description, question type, and question examples thoroughly and make sure you understand the scene in the image and the ability demanded by the question type without copy the example questions

  73. [75]

    Analyze the special color pattern of the image and generate a modality description to explain what the color filter is trying to emphasize

  74. [76]

    Make a strategy to create challenging questions about the image and its color filter to meet the requirements as below to test the ability of the subject (a visual assistant)

  75. [77]

    Your question should be clear and meaningful, requiring the subject having a strong ability that connecting common knowledge to the modality description to solve the question

    Create one question of the given question type based on the strategy and image according to the modality description. Your question should be clear and meaningful, requiring the subject having a strong ability that connecting common knowledge to the modality description to solve the question

  76. [78]

    The answer should be clear and distinct

    Provide the correct answer to your question according to the image and the modality description. The answer should be clear and distinct

  77. [79]

    Adjust the number of incorrect answers according to your question (i.e

    Generate at most {negative_samples_num} different plausible incorrect answers that the subject might give if it is doesn't understand the the modality description or lacks abilities. Adjust the number of incorrect answers according to your question (i.e. if your question only has'yes'or'no'as answers, only generate one incorrect answer). [REQUIREMENT]

  78. [80]

    Please ensure that the image information and the modality description are fully utilized, and make sure the answer cannot be inferred from the question alone

  79. [81]

    Make the length of the incorrect answers similar to the correct answer to prevent that the correct answer is always the longest one

  80. [82]

    You need to generated all kinds of questions related to the question type and the ability behind the question type

    Create diverse questions according to the question type, but please DO NOT completely copy the content of the example questions. You need to generated all kinds of questions related to the question type and the ability behind the question type. Supplementary Material 23

Showing first 80 references.