Pith. sign in

REVIEW 45 references

Instruction-augmented Multimodal Alignment for Image-Text and Element Matching

T0 review · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned multimodal score model with soft Q-Align scoring, element-conditioned prompts, and self-training on validation pseudo-labels takes first place in NTIRE 2025 Track 1 image-text alignment.

arxiv 2504.12018 v1 pith:EXWV4OXT submitted 2025-04-16 cs.CV

Xinli Yue , JianHui Sun , Junda Lu , Liangchao Yao , Fan Xia , Tianyi Wang , Fengyun Rao , Jing Lyu
show 1 more author
Yuetang Deng
This is my paper · ORCID
classification cs.CV
keywords alignmentimage-textaugmentationelementmodelmatchingmultimodalstrategy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-image models produce pictures from prompts, but judging whether the picture really matches the prompt is still hard. iMatch trains a large vision-language model to output a score from 1 to 5 for the overall match and separate scores for individual elements in the prompt, for example 'a red umbrella' or 'two people'. It does this by turning scores into letter labels and using the model's word probabilities to get a continuous number instead of a coarse integer.

The paper adds several tricks during training: perturbing element labels, adding the prompt type, augmenting images with brightness, cropping, and distortion, and embedding element scores into the question prompt. At test time the model first predicts element scores, then uses those predictions as extra inputs when scoring the whole image-text pair. The authors also take their own model's predictions on the validation set, treat them as labels, and retrain on the combined data.

On the NTIRE 2025 challenge leaderboard, their system placed first. On the EvalMuse-40K validation set, they report higher correlations than previous methods such as FGA-BLIP2. One caveat: the validation set was used both to make pseudo-labels and to report the headline correlation numbers, so part of that reported gain may come from the model having seen validation data during retraining.

Extended reading notes

Core claim

Section 4.2, Table 1: 'iMatch achieves significant gains in both SRCC and PLCC, outperforming all baselines' with SRCC 0.8304 and PLCC 0.8294 on the EvalMuse-40K validation set, and Table 3: 'Our iMatch ranked first, outperforming all competitors across all key metrics' in the NTIRE 2025 Track 1 challenge. If true, the method is a state-of-the-art automatic proxy for human image-text alignment judgments.

Load-bearing premise

The paper assumes that the validation set can contribute pseudo-labels to the training set (Section 3.2.2, Eq. 8) and then still serve as an independent evaluation set for the headline numbers in Tables 1 and 2. If the validation set influences training through the pseudo-label loss Lpseudo, the reported 0.8304/0.8294 SRCC/PLCC and 0.8284 ACC are not clean out-of-sample measurements, and the claimed margin over baselines is partly attributable to this reuse rather than to the method's generalization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests on the EvalMuse human annotations as ground truth, on the NTIRE test set as independent, and on the utility of model-generated pseudo-labels. There are no invented physical entities. The main free choices are the element threshold, perturbation magnitude, image augmentation ranges, and standard training hyperparameters; these are not derived and are only partially reported.

free parameters (4)
  • Element hit threshold tau = 3
    Used in Eq. 2 to convert 7-level element predictions into binary hits; the value 3 is chosen by hand rather than derived, and the element ACC in Table 2 is sensitive to it.
  • Score perturbation magnitude epsilon = not specified
    Introduced in Eq. 17 for element label perturbation; its value is not reported, so the exact training noise is unreproducible.
  • Image augmentation ranges and subset probability = alpha~U(0.1,0.5), beta~U(0.2,0.8), gamma~U(0.1,0.5), 10% subset
    Hand-chosen distributions and selection rate in Section 3.2.4; no sensitivity analysis is given, so their reported contribution is conditional on these choices.
  • LoRA rank and learning rate = rank 16, lr 4e-5
    Training hyperparameters in Section 4.1 chosen for all models; standard, but no tuning analysis is provided.
assumptions (5)
  • domain assumption EvalMuse-40K human annotations are reliable ground truth for image-text alignment
    The entire fine-tuning and evaluation in Section 4 relies on these labels; if they are noisy or biased, the reported correlations do not measure alignment.
  • domain assumption MLLM softmax probabilities over rating letters provide a meaningful continuous score distribution
    QAlign augmentation in Section 3.2.1 assumes the closed-set logits after fine-tuning are calibrated enough for weighted averaging to improve over discrete labels.
  • domain assumption Pseudo-labels from the model on the validation set are good enough to serve as training targets
    Section 3.2.2 constructs a pseudo-label loss from the model's own predictions without confidence filtering; if these labels are systematically wrong, retraining will reinforce errors.
  • domain assumption The NTIRE 2025 challenge provides a fair external test set with no overlap with the pseudo-label training data
    The leaderboard result in Table 3 is the main independent evidence; if the test set were contaminated, the claim would collapse.
  • standard math Standard statistical definitions of SRCC, PLCC, and accuracy
    Used for all evaluation tables in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-augmented Multimodal Alignment for Image-Text and Element Matching." pith.science (2026). https://pith.science/paper/EXWV4OXT

@misc{pith2026250412018,
  author       = {Pith},
  title        = {Pith review of: Instruction-augmented Multimodal Alignment for Image-Text and Element Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXWV4OXT}},
  note         = {Machine review of arXiv:2504.12018}
}
read the original abstract

With the rapid advancement of text-to-image (T2I) generation models, assessing the semantic alignment between generated images and text descriptions has become a significant research challenge. Current methods, including those based on Visual Question Answering (VQA), still struggle with fine-grained assessments and precise quantification of image-text alignment. This paper presents an improved evaluation method named Instruction-augmented Multimodal Alignment for Image-Text and Element Matching (iMatch), which evaluates image-text semantic alignment by fine-tuning multimodal large language models. We introduce four innovative augmentation strategies: First, the QAlign strategy creates a precise probabilistic mapping to convert discrete scores from multimodal large language models into continuous matching scores. Second, a validation set augmentation strategy uses pseudo-labels from model predictions to expand training data, boosting the model's generalization performance. Third, an element augmentation strategy integrates element category labels to refine the model's understanding of image-text matching. Fourth, an image augmentation strategy employs techniques like random lighting to increase the model's robustness. Additionally, we propose prompt type augmentation and score perturbation strategies to further enhance the accuracy of element assessments. Our experimental results show that the iMatch method significantly surpasses existing methods, confirming its effectiveness and practical value. Furthermore, our iMatch won first place in the CVPR NTIRE 2025 Text to Image Generation Model Quality Assessment - Track 1 Image-Text Alignment.

Figures

Figures reproduced from arXiv: 2504.12018 by the authors.

Figure 1
Figure 1. The instruction set augmentation process of the proposed iMatch. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The construction process of the element instruction set [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The inference process of the image-text matching aug [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The process of validation set augmentation. We use the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Examples of image augmentation. We employ three [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Model Ensemble. We ensemble five image-text match [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 13 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Kandinsky 3.0 technical report

    Vladimir Arkhipkin, Andrei Filatov, Viacheslav Vasilev, Anastasia Maltseva, Said Azizov, Igor Pavlov, Julia Aga- fonova, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky 3.0 technical report. arXiv preprint arXiv:2312.03511, 2023. 1, 3

    arXiv 2023

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1, 3

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 1, 3, 6, 7

    arXiv 2025

  5. [5]

    blackforestlabs. Flux1.1. https : / / blackforestlabs.ai/, 2024. 1, 3

    work page 2024

  6. [6]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 1, 3, 6, 7

    arXiv 2024

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR,

  8. [8]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 6

    arXiv 2010

Show all 45 references
  1. [9]

    Dreamina

    DreaminaAI. Dreamina. https : / / dreamina . capcut.com/, 2023. 1, 3

    2023

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In ICML, 2024. 1

    2024

  3. [11]

    Ranni: Taming text-to-image diffu- sion for accurate instruction following

    Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffu- sion for accurate instruction following. arXiv preprint arXiv:2311.17002, 2023. 3

    2023 arXiv

  4. [12]

    Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text- to-image generation model evaluation

    Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, et al. Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text- to-image generation model evaluation. arXiv preprint arX...

    2024 arXiv

  5. [13]

    NTIRE 2025 challenge on text to image generation model quality assess- ment

    Shuhao Han, Haotian Fan, Fangyuan Kong, Wenjie Liao, Chunle Guo, Chongyi Li, Radu Timofte, et al. NTIRE 2025 challenge on text to image generation model quality assess- ment. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops ,

    2025

  6. [14]

    Mars: Mixture of auto-regressive mod- els for fine-grained text-to-image synthesis

    Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive mod- els for fine-grained text-to-image synthesis. arXiv preprint arXiv:2407.07614, 2024. 3

    2024 arXiv

  7. [15]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  8. [16]

    Midjourney

    David Holz. Midjourney. https://www.midjourney. com, 2023. 1, 3

    2023

  9. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 6

    2022

  10. [18]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023. 1, 3, 7

    2023

  11. [19]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS, 2023. 1, 3, 6, 7

    2023

  12. [20]

    Evaluating and improving composi- tional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving composi- tional text-to-visual generation. In CVPR, 2024. 6, 7

    2024

  13. [21]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2. 5: Three in- sights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. 1, 3

    2024 arXiv

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  15. [23]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 3

    2023

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024

    2024

  18. [26]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024. 7

    2024

  19. [27]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  20. [28]

    Ovis: Structural embed- ding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 1, 3, 6, 7

    2024 arXiv

  21. [29]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 3

    2021

  22. [30]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  23. [31]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022

    2022

  24. [32]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV, 2024. 1, 3

    2024

  25. [33]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. InNeurIPS,

  26. [34]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 3

    2023 arXiv

  27. [35]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint,

  28. [36]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. In NeurIPS, 2022. 2, 5

    2022

  29. [37]

    Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, et al. Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings. arXiv preprint arXiv...

    2024 arXiv

  30. [38]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 3

    2023 arXiv

  31. [39]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels. In ICML,

  32. [40]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  33. [41]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. In NeurIPS, 2023. 1, 3, 6, 7

    2023

  34. [42]

    What you see is what you read? improving text- image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text- image alignment evaluation. In NeurIPS, 2023. 3, 7

    2023

  35. [43]

    mplug- owl3: Towards long image-sequence understanding in multi- modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug- owl3: Towards long image-sequence understanding in multi- modal large language models. In ICLR, 2024. 7

    2024

  36. [44]

    Swift:a scal- able lightweight infrastructure for fine-tuning, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yun- lin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scal- able lightweight infrastructure for fine-tuning, 2024. 6

    2024

  37. [45]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.