Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Refined image captions injected into web text beat text-only G-SEO, raising visibility by 1.85% (unimodal) and 1.09% (multimodal).

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:44 UTC pith:JCEWZEYI

load-bearing objection Nice first stab at multimodal G-SEO, but the 'multimodal' experiment never actually shows the GSE an image — the headline claim is unsupported. the 3 major comments →

arxiv 2511.04080 v4 pith:JCEWZEYI submitted 2025-11-06 cs.IR

Caption Injection for Optimization in Generative Search Engine

Xiaolu Chen , Jie Bao , Haojie Wu , Zhen Chen , Yong Liao This is my paper
classification cs.IR
keywords caption injectiongenerative search engine optimizationmultimodal retrieval-augmented generationimage captioningprompt engineeringsubjective visibilityG-EvalMRAMG benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the images accompanying a web page carry semantic information that text-only generative-search optimization leaves untapped. The authors propose Caption Injection, a three-stage pipeline that captions an image with a vision-language model, refines that caption by aligning it with the source text's objects, actions, and scene, and then instructs an LLM to insert the refined caption into the source text at the most relevant location. On the MRAMG multimodal benchmark, they report that this injection raises subjective visibility—how often users' attention lands on the content in a generative search answer—by 1.85% relative improvement in a unimodal setting and 1.09% in a multimodal setting, beating all text-only baselines. The paper positions this as the first multimodal G-SEO method and argues that visual semantics, even when projected into text, give web content a measurable edge in generative search.

Core claim

The paper introduces multimodal G-SEO and a method, Caption Injection, that converts images into refined captions and inserts them into source text. The central discovery: a three-stage pipeline—VLM generates an object-action-scene caption, an LLM rewrites it while preserving syntactic structure, then inserts it at the contextually best point—consistently raises relative subjective visibility on the MRAMG benchmark, outperforming all text-only baselines. Reported gains: +1.85% unimodal, +1.09% multimodal. Ablations show the refinement step is necessary, and all methods struggle more in multimodal settings, suggesting cross-modal G-SEO is intrinsically harder.

What carries the argument

The central machinery is the Caption Injection pipeline, a pure prompt-engineering chain with three stages. Structural Generation: a vision-language model produces a caption covering object, action, and scene. Alignment Refinement: an LLM rewrites that caption by extracting source-text fragments matched to those three elements and expanding the caption accordingly, while keeping the original caption's syntactic skeleton intact—this is argued to preserve the LLM's learned attention distribution over semantic entities. Semantic Injection: the LLM is prompted to insert only the refined caption into the source text at a contextually optimal location, with strict instructions not to modify or del

Load-bearing premise

The paper assumes a multimodal GSE turns images into captions before generating an answer; if a real engine reasons over image pixels directly, the measured multimodal gain may not transfer.

What would settle it

Run the same MRAMG benchmark through a multimodal GSE that actually receives raw image pixels (e.g., a multimodal LLM fed image tokens) instead of captions; if the +1.09% relative visibility gain disappears or reverses, the paper's multimodal result is an artifact of its text simulation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Content owners with images can apply this prompt-based pipeline today to improve how often their content is cited or elaborated in generative search answers, with no access to the search engine's internals.
  • The ablation result—refined captions outperform both raw alt-text and VLM-generated captions—implies that simply having an image is not enough; the caption must be textually aligned with the page's content.
  • The uniform drop in all methods' improvement from unimodal to multimodal GSE settings indicates that multimodal visibility optimization is a harder problem, motivating deeper cross-modal fusion research.
  • Near-zero gains on long-text sources (MRAMG-Manual) show that caption injection alone does not address long-context optimization; a separate mechanism is needed for such pages.
  • Because the effect is measured via G-Eval 2.0's averaged sub-scores, changes in any one visibility dimension could be optimized specifically by reweighting the refinement prompt—though the paper reports only the average.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The paper's multimodal GSE simulation replaces images with captions (text + captions + LLM), so the reported multimodal gain measures caption-mediated transfer only—not reasoning over raw pixels. A real engine that consumes image tokens directly could yield a different benefit, a possibility the paper leaves untested.
  • Editorial extension: The ablation shows VLM-generated captions can beat refined ones on specific sub-scores (e.g., click-follow likelihood), implying refinement trades off visibility dimensions. A content creator could tune the refinement prompt to favor a particular G-Eval sub-score rather than the average.
  • Editorial extension: The paper's claim that preserving syntactic structure protects LLM attention predicts that rephrasing the same semantic content with different syntax should reduce the gain; this is a testable separation between semantic alignment and structural preservation that the paper does not perform.
  • Editorial extension: Because the pipeline is text-based, it could generalize to other modalities (audio, video) that can be transcribed into captions; that generalization is not explored in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Caption Injection, described as the first multimodal G-SEO method. A VLM generates a structural caption from each image, an LLM refines it against the source text, and the refined caption is injected into the textual content through prompt engineering. The method is evaluated on the MRAMG benchmark under two simulated GSE configurations: a text-only (unimodal) pipeline and a multimodal pipeline that feeds the GSE text plus image captions. Using G-Eval 2.0 as the evaluation metric, the paper reports relative improvements in subjective visibility of +1.85% (unimodal) and +1.09% (multimodal) over text-only baselines, and concludes that multimodal information benefits G-SEO.

Significance. If the claims are substantiated, the paper opens a useful direction by extending G-SEO from purely textual optimization to a setting where visual semantics are projected into text. The three-stage pipeline (structural generation, alignment refinement, semantic injection) is clearly described, and the authors make code available. However, the central empirical claims rest on an evaluation design that uses a text-only surrogate for a multimodal GSE, an LLM-based metric from the authors' own prior work, and no statistical validation. The reported gains are small (1–2% relative), so the significance currently depends on assumptions that are not fully disclosed or tested.

major comments (3)
  1. [§IV-A1 (Generative Search Engine Simulation)] The multimodal GSE simulation feeds the model only 'text content + image captions + LLM', not original images or a multimodal LLM. The authors say they compared this configuration with 'text content + original images + multimodal LLM' and that the former 'generally yields higher response quality', but the supporting comparison is not shown. This is load-bearing: the claimed multimodal benefit (+1.09%) could simply reflect that adding caption text improves a text-only GSE. The paper does not demonstrate that Caption Injection helps when the GSE can reason over images directly, which is the realistic multimodal scenario claimed in the title and abstract. Please include the omitted comparison or evaluate with a genuine multimodal GSE.
  2. [§IV-A4 (Evaluation Metrics)] The sole evaluation metric is G-Eval 2.0, which is introduced in the authors' own prior work [8]. There is no human validation, no correlation analysis with human judgments, no error bars, and no significance tests. Several dimensions—especially content volume, click-follow likelihood, and positional salience—by construction favor enriched text, so part of the measured gain may be metric-induced. Given the small effect sizes (+1.85% and +1.09%), the possibility that the differences are within LLM-judge noise is not addressed. Please add a human evaluation on a sample, significance testing, or at least a robustness analysis with multiple judge models and seeds.
  3. [§IV-B2 (Ablation Study) and Table V] The text states that 'the refined captions consistently yield higher subjective visibility scores than both the original and VLM-generated captions'. Table V shows average scores of 1.18 for the original caption, 1.11 for the rewritten (refined) caption, and 1.06 for the VLM-generated caption. The refined caption is not the top performer; the original caption is. This contradicts the ablation claim and weakens the argument that the refinement stage is beneficial. Please correct the claim or the table, and clarify what is being compared.
minor comments (5)
  1. [§IV-B1, Table IV] The method names in Table IV are abbreviated inconsistently ('tran seo', 'flue expr', 'quat addi', 'stat addi', 'capt addi') and the table has no descriptive caption. Please use full method names or define all abbreviations.
  2. [§IV-A4, improvement formula] The formula for relative improvement uses 'imprs(r)' in the denominator and 'impression_s(r)' in the numerator, which is typographically inconsistent. Clarify notation and check whether the '+1' term is intentional.
  3. [§IV-B2, Table V] The table title says 'EVALUATION OF G-SEO METHODS ON DIFFERENT METRICS (VALUES×100)', but this is an ablation of caption types. The column header 'Sub.Posi.' and 'Sub.Volu.' are not expanded. Also 'rewriten' is misspelled in the last row.
  4. [§III-B, prompt tables] The prompt for caption injection in Table III is labeled 'Prompt for caption refinement', which duplicates the label of Table II. Please correct the label.
  5. [General] The paper relies heavily on the authors' own unpublished prior work [8] for G-Eval 2.0 and the overall evaluation protocol; please make the dependency explicit in the methodology section so readers can assess the lineage of the metric.

Circularity Check

2 steps flagged

Evaluation metric and multimodal simulation are co-designed with the method, making part of the claimed gain construct-internal.

specific steps
  1. self citation load bearing [Section IV-A4 (Evaluation Metrics); reference [8]]
    "To objectively assess how different G-SEO strategies enhance the subjective visibility of content sources, we adopt G-Eval 2.0 [8] as the subjective evaluation framework, which is specifically adapted for the G-SEO scenario. ... G-Eval 2.0 [8], an enhanced version tailored for G-SEO evaluation, refines this framework by assessing content sources along seven subjective dimensions: relevance, fluency, diversity, uniqueness, click-follow likelihood, positional salience, and content volume."

    G-Eval 2.0 is the authors' own prior work (Chen et al., arXiv:2508.11158), cited as the authoritative measure of the very construct the paper claims to improve. One of its seven dimensions is 'content volume,' and the injection prompt requires 'Do not delete or modify any other part of the given source... do not add anything other than the given text' (Table III), so the intervention is a strict text addition. The metric therefore rewards the method's primary operation by construction. The central +1.85%/+1.09% gains are measured on a rubric designed by the same authors that contains a dimension the method mechanically increases; the evaluation is not an external, independent check.

  2. self definitional [Section IV-A1 (Generative Search Engine Simulation)]
    "By comparatively analyzing generation-level precision, recall, relevance, and fluency across representative MRAG benchmarks, we observe that the configuration using 'text content + image captions + LLM' generally yields higher response quality than that using 'text content + original images + multimodal LLM'. Based on this analysis, we adopt the former configuration in our multimodal GSE simulation."

    The 'multimodal GSE' is simulated as a text LLM that receives image captions as text, not as a model that perceives images. Caption Injection is precisely the operation of extracting captions and inserting them into the source text. The environment is therefore defined by the same caption-injection transformation that is being evaluated: the method turns text into text+captions, and the environment is selected because text+captions 'generally yields higher response quality.' The reported multimodal gain is a consequence of this environment definition; it does not test whether Caption Injection helps a GSE that can reason over the original images. The comparison that would break the circularity is not shown.

full rationale

The method itself is not derived from the metric in the sense of fitted parameters, and the paper does compare against text-only baselines in the same simulated environment, so a score of 10 would be too harsh. However, two load-bearing choices are construct-internal. First, the evaluation metric G-Eval 2.0 is a self-citation from the same authors, and its 'content volume' dimension is automatically increased by Caption Injection's instruction never to delete or modify text, only to insert captions. Second, the 'multimodal GSE' simulation is defined to use text+image-captions rather than original images, which is exactly the input format Caption Injection produces; the paper's own observation that this configuration 'generally yields higher response quality' is the premise being tested, not independent evidence. Together these make the headline improvement partially an artifact of the paper's own evaluation choices. The reported unimodal gain is a more direct empirical result, but the multimodal claim and the metric's role both exhibit partial circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted numeric parameters; the method is an empirical pipeline whose claims rest on the evaluation metric and the GSE simulation assumptions above.

axioms (4)
  • domain assumption G-Eval 2.0 scores on seven dimensions are a valid measure of subjective visibility of content sources.
    Section IV-A4 adopts G-Eval 2.0 from the authors' prior work [8] without human agreement; all conclusions are relative to this metric.
  • domain assumption A GSE simulated as 'text content + image captions + LLM' is representative of real multimodal GSEs.
    Section IV-A1 justifies this by an unreported comparative analysis; this configuration reduces multimodal input to text, and the paper's conclusions depend on it.
  • domain assumption GLM-4-9B has low hallucination and is a stable representative backbone for GSE generation.
    Section IV-A1 relies on an external hallucination leaderboard as evidence; generation quality of other backbones is not tested.
  • ad hoc to paper Existing MRAMG captions are incomplete or superficial, so structural generation and refinement are needed.
    Section III-B1 asserts this observation to motivate the pipeline; no quantitative evidence of caption quality is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 13067 in / 10562 out tokens · 97243 ms · 2026-08-03T23:44:53.092355+00:00 · methodology

0 comments
read the original abstract

Generative Search Engine (GSE) leverages the Retrieval-Augmented Generation (RAG) technique and the Large Language Model (LLM) to integrate multi-source information and provide users with accurate and comprehensive responses. Unlike traditional search engines that present results in ranked lists, GSE shifts users' attention from sequential browsing to content-driven subjective perception, not only driving a paradigm shift in information retrieval but also highlighting the importance of enhancing the subjective visibility of content in generative search. In this context, Generative Search Engine Optimization (G-SEO) methods have emerged as a new research focus. With the rapid advancement of Multimodal Retrieval-Augmented Generation (MRAG) techniques, GSE can now efficiently integrate text, images, audio, and video, producing richer responses that better satisfy complex information needs. Existing G-SEO methods, however, remain limited to text-based optimization and fail to fully exploit multimodal data. To address this gap, we propose Caption Injection, the first multimodal G-SEO approach, which extracts captions from images and injects them into textual content, integrating visual semantics to enhance the subjective visibility in generative search. We systematically evaluate Caption Injection on MRAMG, a benchmark for MRAG, under both unimodal and multimodal settings. Experimental results show that Caption Injection significantly outperforms text-only G-SEO baselines under the G-EVAL metric, effectively improving the subjective visibility of content perceived by users, and demonstrating the practical benefits of multimodal information in G-SEO. The source code for this work is openly available at https://github.com/GrayChan04/Caption-Injection.

Figures

Figures reproduced from arXiv: 2511.04080 by Haojie Wu, Jie Bao, Xiaolu Chen, Yong Liao, Zhen Chen.

Figure 1
Figure 1. Figure 1: Comparison of result presentation across different types of search engines. Traditional search engines (blue section on the left) display retrieved web [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the Caption Injection pipeline. The image of the web [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 6 linked inside Pith

  1. [1]

    What is search engine optimization: Seo?

    N. Yalc ¸ın and U. K ¨ose, “What is search engine optimization: Seo?” Procedia - Social and Behavioral Sciences, vol. 9, pp. 487–493, 2010, world Conference on Learning, Teaching and Administration Pa- pers. [Online]. Available: https://www.sciencedirect.com/science/article/ pii/S1877042810022901

    2010

  2. [2]

    Search engine optimization: A review,

    F. ALMUKHTAR, N. MAHMOODD, and S. KAREEM, “Search engine optimization: A review,”Applied Computer Science, vol. 17, no. 1, p. 70–80, Mar. 2021. [Online]. Available: https://ph.pollub.pl/index.php/ acs/article/view/3098

    2021

  3. [3]

    Geo: Generative engine optimization,

    P. Aggarwal, V . Murahari, T. Rajpurohit, A. Kalyan, K. Narasimhan, and A. Deshpande, “Geo: Generative engine optimization,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 5–16. [Online]. Available: https://doi. org/10.1145/3637528.3671900

    arXiv 2024

  4. [4]

    Beyond keywords: Driving generative search engine optimization with content-centric agents,

    Q. Chen, J. Chen, H. Huang, Q. Shao, J. Chen, R. Hua, H. Xu, R. Wu, R. Chuan, and J. Wu, “Beyond keywords: Driving generative search engine optimization with content-centric agents,” 2025. [Online]. Available: https://arxiv.org/abs/2509.05607

    arXiv 2025

  5. [5]

    Ranking manipulation for conversational search engines,

    S. Pfrommer, Y . Bai, T. Gautam, and S. Sojoudi, “Ranking manipulation for conversational search engines,” inProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, Y . Al- Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 9523–9552. [Online]. Availabl...

    2024

  6. [6]

    Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,” inProceedings of the 16th ACM Workshop on Artificial Intelligence and Security, ser. AISec ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 79–90...

    arXiv 2023

  7. [7]

    Manipulating large language models to increase product visibility,

    A. Kumar and H. Lakkaraju, “Manipulating large language models to increase product visibility,” 2024. [Online]. Available: https://arxiv.org/ abs/2404.07981

    Pith/arXiv arXiv 2024

  8. [8]

    Role- augmented intent-driven generative search engine optimization,

    X. Chen, H. Wu, J. Bao, Z. Chen, Y . Liao, and H. Huang, “Role- augmented intent-driven generative search engine optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2508.11158

    arXiv 2025

  9. [9]

    Beyond seo: A transformer- based approach for reinventing web content optimisation,

    F. L ¨uttgenau, I. Colic, and G. Ramirez, “Beyond seo: A transformer- based approach for reinventing web content optimisation,” 2025. [On- line]. Available: https://arxiv.org/abs/2507.03169

    Pith/arXiv arXiv 2025

  10. [10]

    An effective seo techniques and technologies guide-map,

    K. I. Roumeliotis and N. D. Tselikas, “An effective seo techniques and technologies guide-map,”Journal of Web Engineering, vol. 21, no. 5, pp. 1603–1649, July 2022

    2022

  11. [11]

    Python driven keyword analysis for seo optimization,

    A. P. Kanara, P. Kumari, and B. R. Prathap, “Python driven keyword analysis for seo optimization,” in2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS), vol. 1, March 2024, pp. 1170–1176

    2024

  12. [12]

    Autotrendykeywords: Real-time ai-driven trend-based seo using llms,

    P. Vadlapati, “Autotrendykeywords: Real-time ai-driven trend-based seo using llms,” 2024

    2024

  13. [13]

    Large language models for search engine optimization in e-commerce,

    G. Chodak and K. Bła ˙zyczek, “Large language models for search engine optimization in e-commerce,” inAdvanced Computing, D. Garg, J. J. P. C. Rodrigues, S. K. Gupta, X. Cheng, P. Sarao, and G. S. Patel, Eds. Cham: Springer Nature Switzerland, 2024, pp. 333–344

    2024

  14. [14]

    Search engine optimization by using machine learning for web page classification,

    S. S. Shaffi and I. Muthulakshmi, “Search engine optimization by using machine learning for web page classification,” in2022 Interna- tional Conference on Augmented Intelligence and Sustainable Systems (ICAISS), Nov 2022, pp. 342–349

    2022

  15. [15]

    A page rank-based analytical design of effective search engine optimization,

    V . Srinivas and P. Gowda, “A page rank-based analytical design of effective search engine optimization,”Iaes International Journal of Artificial Intelligence (Ij-Ai), vol. 14, no. 1, pp. 73–82, 2025

    2025

  16. [16]

    Automatic document editing for improved ranking,

    N. Bardas, T. Mordo, O. Kurland, and M. Tennenholtz, “Automatic document editing for improved ranking,” inProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 2779–2783. [Online]. Available: https://doi.org/10.1145/...

    arXiv 2025

  17. [17]

    Adversarial search engine optimization for large language models,

    F. Nestaas, E. Debenedetti, and F. Tram `er, “Adversarial search engine optimization for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18382

    Pith/arXiv arXiv 2024

  18. [18]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, Feb 2019

    2019

  19. [19]

    Vision + x: A survey on multimodal learning in the light of data,

    Y . Zhu, Y . Wu, N. Sebe, and Y . Yan, “Vision + x: A survey on multimodal learning in the light of data,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 9102–9122, Dec 2024

    2024

  20. [20]

    Multimodal learning with transform- ers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transform- ers: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 113–12 132, Oct 2023

    2023

  21. [21]

    Multi-modal retrieval augmented multi-modal generation: Datasets, evaluation metrics and strong baselines,

    Z.-A. Ma, T. Lan, R.-C. Tu, Y . Hu, Y .-S. Zhu, T. Zhang, H. Huang, Z. Wu, and X.-L. Mao, “Multi-modal retrieval augmented multi-modal generation: Datasets, evaluation metrics and strong baselines,” 2025. [Online]. Available: https://arxiv.org/abs/2411.16365

    Pith/arXiv arXiv 2025

  22. [22]

    Mramg- bench: A comprehensive benchmark for advancing multimodal retrieval- augmented multimodal generation,

    Q. Yu, Z. Xiao, B. Li, Z. Wang, C. Chen, and W. Zhang, “Mramg- bench: A comprehensive benchmark for advancing multimodal retrieval- augmented multimodal generation,” inProceedings of the 48th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’25. New York, NY , USA: Association for Computing Machinery, 2...

    arXiv 2025

  23. [23]

    MuRAR: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering,

    Z. Zhu, D. Lee, H. Zhang, S. Sree Harsha, L. Feujio, A. Maharaj, and Y . Li, “MuRAR: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering,” inProceed- ings of the 31st International Conference on Computational Linguistics: System Demonstrations, O. Rambow, L. Wanner, M. Apidianaki, H. Al- Khalifa, B...

    2025

  24. [24]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, ...

    2021

  25. [25]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inProceedings of the 39th International Conference on Machine Learn- ing, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol

  26. [26]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language mod- els,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language mod- els,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol

  27. [27]

    The revolution of multimodal large language models: A survey,

    D. Caffagni, F. Cocchi, L. Barsellotti, N. Moratelli, S. Sarto, L. Baraldi, L. Baraldi, M. Cornia, and R. Cucchiara, “The revolution of multimodal large language models: A survey,” inFindings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics...

    2024

  28. [28]

    Unified multimodal understanding and generation models: Advances, challenges, and oppor- tunities,

    X. Zhang, J. Guo, S. Zhao, M. Fu, L. Duan, J. Hu, Y . X. Chng, G.-H. Wang, Q.-G. Chen, Z. Xu, W. Luo, and K. Zhang, “Unified multimodal understanding and generation models: Advances, challenges, and oppor- tunities,” 2025. [Online]. Available: https://arxiv.org/abs/2505.02567

    arXiv 2025

  29. [29]

    Pre-trained language models for text generation: A survey,

    J. Li, T. Tang, W. X. Zhao, J.-Y . Nie, and J.-R. Wen, “Pre-trained language models for text generation: A survey,”ACM Comput. Surv., vol. 56, no. 9, Apr. 2024. [Online]. Available: https://doi.org/10.1145/ 3649449

    2024

  30. [30]

    Altogether: Image captioning via re-aligning alt-text,

    H. Xu, P.-Y . Huang, X. Tan, C.-F. Yeh, J. Kahn, C. Jou, G. Ghosh, O. Levy, L. Zettlemoyer, W.-t. Yih, S.-W. Li, S. Xie, and C. Fe- ichtenhofer, “Altogether: Image captioning via re-aligning alt-text,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida,...

    2024

  31. [31]

    Fuse- Cap: Leveraging Large Language Models for Enriched Fused Image Captions ,

    N. Rotstein, D. Bensaid, S. Brody, R. Ganz, and R. Kimmel, “ Fuse- Cap: Leveraging Large Language Models for Enriched Fused Image Captions ,” in2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Los Alamitos, CA, USA: IEEE Computer Society, Jan. 2024, pp. 5677–5688. [Online]. Available: https://doi. ieeecomputersociety.org/10.1109/...

    arXiv 2024

  32. [32]

    Im- proving clip training with language rewrites,

    L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y . Tian, “Im- proving clip training with language rewrites,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 35 544–35 575. [On- line]. Available: https://proceedings.neurips.cc/paper fi...

    2023

  33. [33]

    CapsFusion: Rethinking Image-Text Data at Scale ,

    Q. Yu, Q. Sun, X. Zhang, Y . Cui, F. Zhang, Y . Cao, X. Wang, and J. Liu, “ CapsFusion: Rethinking Image-Text Data at Scale ,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2024, pp. 14 022–14 032. [Online]. Available: https://doi.ieeecomputersociety. org/10.1109/CVPR52733....

    arXiv 2024

  34. [34]

    Veclip: Improving clip training via visual-enriched captions,

    Z. Lai, H. Zhang, B. Zhang, W. Wu, H. Bai, A. Timofeev, X. Du, Z. Gan, J. Shan, C.-N. Chuah, Y . Yang, and M. Cao, “Veclip: Improving clip training via visual-enriched captions,” inComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–Octo- ber 4, 2024, Proceedings, Part XLII. Berlin, Heidelberg: Springer- Verlag, 2024, p. 111–...

    2024

  35. [35]

    Enhancing inter- active image retrieval with query rewriting using large language models and vision language models,

    H. Zhu, J.-H. Huang, S. Rudinac, and E. Kanoulas, “Enhancing inter- active image retrieval with query rewriting using large language models and vision language models,” inProceedings of the 2024 International Conference on Multimedia Retrieval, ser. ICMR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 978–987. [Online]. Available: h...

    arXiv 2024

  36. [36]

    Leveraging Large Lan- guage Models for Multimodal Search ,

    O. Barbany, M. Huang, X. Zhu, and A. Dhua, “ Leveraging Large Lan- guage Models for Multimodal Search ,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2024, pp. 1201–

    2024

  37. [37]

    MuRAG: Multi- modal retrieval-augmented generator for open question answering over images and text,

    W. Chen, H. Hu, X. Chen, P. Verga, and W. Cohen, “MuRAG: Multi- modal retrieval-augmented generator for open question answering over images and text,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y . Goldberg, Z. Kozareva, and Y . Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Lingui...

    2022

  38. [38]

    Cross-modality pyramid alignment for visual intention understanding,

    M. Ye, Q. Shi, K. Su, and B. Du, “Cross-modality pyramid alignment for visual intention understanding,”IEEE Transactions on Image Pro- cessing, vol. 32, pp. 2190–2201, 2023

    2023

  39. [39]

    Uncertain multimodal intention and emotion understanding in the wild,

    Q. Yang, Q. Shi, T. Wang, and M. Ye, “Uncertain multimodal intention and emotion understanding in the wild,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025, pp. 24 700–24 709

    2025

  40. [40]

    Contextual augmented global contrast for multimodal intent recognition,

    K. Sun, Z. Xie, M. Ye, and H. Zhang, “Contextual augmented global contrast for multimodal intent recognition,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 26 963–26 973

    2024

  41. [41]

    Actions at a glance: The time course of action, object, and scene recognition in a free recall paradigm,

    M. Reger, O. Vrabie, G. V olberg, and A. Lingnau, “Actions at a glance: The time course of action, object, and scene recognition in a free recall paradigm,”Cognitive, Affective, & Behavioral Neuroscience, pp. 1–15, 2025

    2025

  42. [42]

    Zero-shot action recognition from diverse object-scene compositions,

    C. Bretti and P. Mettes, “Zero-shot action recognition from diverse object-scene compositions,”arXiv preprint arXiv:2110.13479, 2021

    Pith/arXiv arXiv 2021

  43. [43]

    Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image,

    D. Driess, J.-S. Ha, and M. Toussaint, “Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image,”arXiv preprint arXiv:2006.05398, 2020

    Pith/arXiv arXiv 2006

  44. [44]

    Context understanding in computer vision: A survey,

    X. Wang and Z. Zhu, “Context understanding in computer vision: A survey,”Comput. Vis. Image Underst., vol. 229, no. C, Mar. 2023. [Online]. Available: https://doi.org/10.1016/j.cviu.2023.103646

    arXiv 2023

  45. [45]

    G-eval: NLG evaluation using gpt-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: NLG evaluation using gpt-4 with better human alignment,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 2511–2522. [Online]. Available: https:/...

    2023

  46. [162]

    12 888–12 900

    PMLR, 17–23 Jul 2022, pp. 12 888–12 900. [Online]. Available: https://proceedings.mlr.press/v162/li22n.html

    2022

  47. [202]

    19 730–19 742

    PMLR, 23–29 Jul 2023, pp. 19 730–19 742. [Online]. Available: https://proceedings.mlr.press/v202/li23q.html

    2023

  48. [1210]

    Available: https://doi.ieeecomputersociety.org/10.1109/ CVPRW63382.2024.00127

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ CVPRW63382.2024.00127

    arXiv 2024