Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes

Anthony Moore; Antonio Carlos Rivera; Steven Robinson

REVIEW 4 major objections 5 minor 22 references

Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Structured object tags injected into the prompt give the LLM fine-grained visual context, lift benchmarks, and let retrieval run offline.

desk verdict A thin REMIND variant with unsupported SOTA claims and a broken efficiency story; not ready for review. read the letter →

arxiv 2412.11396 v1 pith:NG5YD5VS submitted 2024-12-16 cs.CV

Antonio Carlos Rivera , Anthony Moore , Steven Robinson This is my paper · ORCID

classification cs.CV

keywords Object-AwareReasoningVisionandLanguageLargeVision-LanguageModelsRetrieval-AugmentedPromptingVisualQuestionAnsweringSceneGraphParsingInferenceLatencyFine-Grained

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VRAP tries to establish that large vision-language models can become reliably object-aware without a retrieval module at inference time. It builds a prompt from structured tags—detected objects, their attributes, and pairwise relationships—enriched with external knowledge, and trains the LLM to answer from that enriched text alone. Across VQAv2, GQA, VizWiz, and COCO captioning, the paper reports accuracy and caption scores above BLIP-2, InstructBLIP, and ShareGPT4V, including 73.5 on VQAv2 and 65.8 on GQA. It also reports a 40% reduction in inference latency because tag construction is moved out of the query path. If the claim holds, object-aware visual reasoning becomes cheaper and more interpretable without architectural changes.

What carries the argument

The load-bearing mechanism is the retrieval-augmented tag generator paired with prompt serialization. A visual encoder $E_v$ maps the image to features $f_v$; a scene graph parser $P$ extracts objects, attributes, and relationships; external knowledge is folded in to yield the structured tag set $T$. The prompt is simply $P_{\text{text}} = \text{Concat}(q, \text{`` Tags: ''}, T)$, and the LLM predicts from that text. This mechanism does the paper's work: it turns visual detail into tokens the LLM can condition on, makes the visual context inspectable, and removes any need for retrieval during response generation.

What would settle it

Run VRAP on a test set where tags must be generated on the fly from raw pixels and measure end-to-end latency including visual encoding and scene-graph parsing. If that end-to-end time is not below retrieval-based alternatives, or if accuracy drops when the offline tagger is replaced by a realistic online tagger, the central latency and decoupling claims fail.

Watch

Extended reading notes

Core claim

The central claim is that retrieval-enriched object tags are sufficient to carry object-level visual knowledge into an LLM. The paper defines the tag set as objects $\{o_i\}$, attribute pairs $(o_i, a_i)$, and relational triples $(o_i, r_k, o_j)$, generated by a pretrained visual encoder and scene graph parser, enriched with external knowledge, and serialized as text. Training uses a combined objective of generative loss, contrastive tag-relevance loss, and tag-generation loss; at inference the model only sees the query plus pre-generated tags. On this basis VRAP reports state-of-the-art results on VQAv2, GQA, VizWiz, and COCO, with stronger handling of queries about unseen objects than the strongest baseline. The paper's conclusion is that decoupling retrieval from inference preserves object-awareness while making the pipeline efficient and interpretable.

Load-bearing premise

The whole speed-and-simplicity story assumes that structured tags for every image, including never-before-seen test images, can be produced offline at no inference cost, but the paper never specifies how tags are generated for new images without running the visual encoder and scene graph parser at query time.

Editorial extensions

If this is right

Any instruction-tuned LLM could be made object-aware by feeding it the same serialized tag stream, with no change to the language-model architecture.
Because tags are precomputed, per-query latency is bounded by the LLM call; the paper measures 890 ms per query versus 1250 ms for ShareGPT4V.
On queries about unseen objects, VRAP reports higher accuracy than ShareGPT4V (69.8 vs 62.3 on VQAv2 and 53.5 vs 47.1 on VizWiz), implying tag descriptions substitute for seeing the object during pretraining.
Scaling the training corpus with more tag-rich data raises accuracy further (75.2 VQAv2, 67.3 GQA), so the framework's gains are tied to tag coverage and diversity, not model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported 40% latency saving is only meaningful if tag generation is truly free at inference; a rigorous comparison would count the visual-encoder and scene-graph-parser time on unseen test images, which likely shrinks the gap.
Because the tags are textual and human-readable, a natural stress test is to corrupt, drop, or reorder tags and measure how gracefully the model degrades; this would separate the value of tag content from the value of prompt structure.
The framework suggests a modular path: swap the external-knowledge enrichment for a purely learned tagger, or replace the LLM with a cheaper one, and re-measure; if the tag stream is the carrier of visual knowledge, performance should track tag quality more than model scale.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a REMIND variant with the comparison missing, wrapped in experiments that don't hold together. The core pipeline—extract object tags with a scene-graph parser and prepend them to the prompt—is what arXiv:2406.10839 already does, and the paper cites it without ever benchmarking against it. The contrastive loss is a small extension, but there is no evidence it helps beyond a one-line ablation on VQA v2.

What the paper does well: the writing is clear, the loss decomposition is sensible, and the ablation (tags matter, contrastive loss helps) is the right kind of check. If the experiments were reproducible, this would be a modest incremental result.

The soft spots are load-bearing. First, the SOTA claim in Table 1 is unsupported: BLIP-2 at 41.0 VQAv2 and InstructBLIP at 49.2 are far below published numbers, and no references anchor the baseline results. Second, the closest prior method, REMIND, is absent from the comparison. Third, the efficiency story breaks on inspection. The tags T are produced by the visual encoder Ev and parser P (Eq. 1 and Sec. 3.3). For any new image, those components must run before the prompt is formed. The paper says tags are 'pre-generated' and processed 'offline,' but never says how deployment on unseen images avoids running Ev and P. If tags are precomputed for the fixed benchmark test sets, the method does not generalize; if they are computed at query time, the 'no retrieval' claim collapses and the 890ms table omits a large chunk of computation. And the abstract's 40% reduction contradicts Table 6, which shows 28.8%. Fourth, no code, no named models, no hyperparameters, no error bars. The human evaluation reports no inter-annotator agreement.

None of this is fixable by minor revision. The paper needs a rewrite with real baselines, a REMIND comparison, a timing breakdown, and a story about new images. As it stands, it is not a research contribution; it is a summary of a plausible-sounding project.

Who gets value? Maybe a student looking for a template of a RAG-for-VLM pipeline. That is about it. I would not send this to referees in its current state.

Referee Report

4 major / 5 minor

Summary. The paper proposes VRAP, a framework that enriches prompts of large vision-language models (LVLMs) with structured object tags (objects, attributes, relationships) extracted offline by pretrained visual encoders and scene graph parsers. At inference, these tags are concatenated with the query to form the prompt, and the LLM generates the response without runtime retrieval. Training uses a multitask objective combining a generative loss, a contrastive loss for tag relevance, and an auxiliary tag-generation loss. The authors claim state-of-the-art performance on VQAv2, GQA, VizWiz, and COCO, supported by comparisons against BLIP-2, InstructBLIP, and ShareGPT4V, along with ablations, human evaluation, robustness analysis, scalability analysis, and efficiency measurements. The paper also claims a 40% reduction in inference latency by eliminating runtime retrieval.

Significance. If properly substantiated, the core idea of decoupling tag generation from inference and injecting retrieval-augmented tags into the prompt is a potentially useful direction for object-aware reasoning, and the contrastive learning component for tag relevance is an interesting addition. The claimed efficiency benefit of offline tag generation is also relevant to practical deployment. However, the manuscript in its current form does not provide the evidence needed to verify these contributions: baseline numbers are not anchored to published results or a described evaluation protocol, the inference-time status of tag generation is unexplained, and the closest prior method (REMIND) is not compared. The potential significance is real, but the paper's claims are currently unverified.

major comments (4)

[Section 4.2, Table 1] The state-of-the-art claim is unsupported. Table 1 reports BLIP-2 at 41.0 VQAv2 accuracy, InstructBLIP at 49.2, and ShareGPT4V at 71.2, but no experimental details are given: no test splits, no model checkpoints or configurations, no evaluation protocol, and no error bars. The reported baseline values are also not compared to the published numbers for these well-known models. Without anchored and reproducible baselines, the table cannot verify the central claim that VRAP (73.5) achieves state-of-the-art performance.
[Section 4.5, Table 6; Section 3.7] The efficiency claim is internally inconsistent and the tag-generation protocol is unexplained. The text states a '40% reduction in inference time' and Table 6 lists a relative speedup of 1.4x, but the reported times (890 ms vs 1250 ms) correspond to a 28.8% reduction, not 40%. More critically, Eq. (9) says inference uses pre-generated tags T, yet the paper never explains how T is produced for evaluation images without running the visual encoder Ev (Eq. 1) and scene graph parser P (Section 3.3). If tags are precomputed for the test set, the method is not a deployable system for arbitrary unseen images; if they are computed at inference, the claimed latency advantage over retrieval-based methods collapses, and no timing breakdown for Ev and P is provided.
[Section 2.2 and Section 4.2] The paper fails to compare against or clearly differentiate from REMIND [2], which is the closest prior work: it retrieves object tags to remind LVLMs of object-aware knowledge. The abstract and Section 1 claim that VRAP differs from prior work relying on multimodal retrieval systems, but REMIND appears to be exactly such an approach. No comparison, no discussion of architectural or objective differences, and no quantitative relation to this prior method are given, preventing the reader from assessing novelty and relative performance.
[Section 4.5, Tables 2-5; Section 4.4] Several evaluation components lack necessary controls and details. The 'unseen objects' results in Table 4 do not describe how 'unseen' is defined or how tags for those images are generated; the human evaluation in Table 3 reports only percentages with no number of annotators, no agreement statistics, and no instructions; the ablation and scalability tables (Tables 2 and 5) do not specify training-data composition, hyperparameter values, or whether the same protocol as in Table 1 was used. These omissions make the secondary claims such as robustness to unseen objects and scalability unverifiable.

minor comments (5)

[Section 3.5, Eq. (5)] The symbol T is used both for the tag set in Eq. (2) and for the sequence length in Eq. (5), which is confusing and should be disambiguated.
[Section 4.1] The introduction and experimental setup mention VisualDialog++, MultiModalQA, CC3M, and CC12M, but no results are reported on these datasets and no explanation is given of their role in training; without this, the term 'original datasets' in Table 5 is undefined.
[Section 4.3, Table 2] The ablation results are reported as single numbers without standard deviations or significance tests, so the size of the component contributions cannot be assessed.
[Section 4.5, Table 7] The qualitative examples are anecdotal and no procedure is described for how they were selected, so they do not provide systematic evidence of improvement.
[Section 2.2] The related-work section does not discuss how the proposed contrastive loss for tag relevance relates to the training objective of REMIND [2], which would help position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VRAP's benchmark numbers are empirical outcomes of a prompt-engineering pipeline, not consequences entailed by its definitions.

full rationale

The paper's derivation chain is a standard prompt-engineering pipeline: a pretrained visual encoder Ev (Eq. 1) and scene graph parser P (Eq. 2) produce tags T, which are concatenated with the query (Eq. 3), and the LLM generates y (Eqs. 4 and 9). The reported benchmark results are empirical measurements, not algebraic consequences of these definitions. No parameter is fitted to a subset of the benchmark and then 'predicted' on a closely related quantity; the losses in Eqs. 5-7 supervise generation, tag relevance, and tag quality, but Table 1 accuracy figures are not derived from these losses by construction. The method closely resembles the prior REMIND work [2], which is explicitly cited and discussed, and the novelty claim of offline tag generation has an unsupported latency accounting (Section 4.5 says '40% reduction' while Table 6 gives 890ms vs 1250ms, a 28.8% reduction), but this is an evidence/consistency flaw rather than a circular step. Citations to Zhou et al. are numerous but not self-citations by the authors (Rivera, Moore, Robinson), and no load-bearing argument reduces to an unverified self-cited theorem. Therefore no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No named visual encoder, scene graph parser, LLM backbone, or external knowledge base is given. The method's success rests entirely on unspecified components and on an offline-tag assumption that is not operationalized.

free parameters (3)

lambda_gen (generative loss weight) = not reported
Balancing coefficient in Eq. 8; no value or search grid is given, yet it controls the training objective.
lambda_contrast (contrastive loss weight) = not reported
Balancing coefficient in Eq. 8; no value given.
lambda_tag (auxiliary tag loss weight) = not reported
Balancing coefficient in Eq. 8; no value given.

assumptions (4)

domain assumption Pretrained visual encoder E_v and scene graph parser P exist and produce accurate, complete tags for all evaluation images.
Section 3.3 relies on these unspecified components; the error analysis in Section 4.5 admits tags are often incomplete.
domain assumption Serialized tags retain enough object-level information for the LLM to reason correctly.
Section 3.4 prompt construction assumes textual tags are the only visual information available to the LLM.
ad hoc to paper Offline tag generation is available for every test image at no inference cost.
Section 3.7 states tags are pre-generated and inference needs no retrieval, but no method is provided to obtain test-image tags without running the tagger.
ad hoc to paper Positive and negative tag sets T+ and T- can be constructed for the contrastive loss.
Section 3.5 defines L_contrast in Eq. 6 without describing how positive and negative examples are mined.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes." pith.science (2026). https://pith.science/paper/NG5YD5VS

@misc{pith2026241211396,
  author       = {Pith},
  title        = {Pith review of: Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NG5YD5VS}},
  note         = {Machine review of arXiv:2412.11396}
}

read the original abstract

Object-aware reasoning in vision-language tasks poses significant challenges for current models, particularly in handling unseen objects, reducing hallucinations, and capturing fine-grained relationships in complex visual scenes. To address these limitations, we propose the Vision-Aware Retrieval-Augmented Prompting (VRAP) framework, a generative approach that enhances Large Vision-Language Models (LVLMs) by integrating retrieval-augmented object tags into their prompts. VRAP introduces a novel pipeline where structured tags, including objects, attributes, and relationships, are extracted using pretrained visual encoders and scene graph parsers. These tags are enriched with external knowledge and incorporated into the LLM's input, enabling detailed and accurate reasoning. We evaluate VRAP across multiple vision-language benchmarks, including VQAv2, GQA, VizWiz, and COCO, achieving state-of-the-art performance in fine-grained reasoning and multimodal understanding. Additionally, our ablation studies highlight the importance of retrieval-augmented tags and contrastive learning, while human evaluations confirm VRAP's ability to generate accurate, detailed, and contextually relevant responses. Notably, VRAP achieves a 40% reduction in inference latency by eliminating runtime retrieval. These results demonstrate that VRAP is a robust and efficient framework for advancing object-aware multimodal reasoning.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

[2]

Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags

Qi, D., Zhao, H., Wei, Z., Li, S.: Reminding multimodal lar ge lan- guage models of object-aware knowledge with retrieved tags . CoRR abs/2406.10839 (2024). https://doi.org/10.48550/ARXIV.2406.10839, https://doi.org/10.48550/arXiv.2406.10839

work page Pith review arXiv doi:10.48550/arxiv.2406.10839 2024
[1]

In: Gold- berg, Y., Kozareva, Z., Zhang, Y

Chen, W., Hu, H., Chen, X., Verga, P., Cohen, W.W.: Murag: M ultimodal retrieval- augmented generator for open question answering over image s and text. In: Gold- berg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2 022 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2022 , Abu Dhabi, United Arab Emirates, December 7-11, 2...

doi:10.18653/v1/2022.emnlp-main.375 2022
[3]

In: Krause, A., Brunskill, E., Cho, K., Enge lhardt, B., Sabato, S., Scarlett, J

Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskove c, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.: Retrieval-augmented m ultimodal lan- guage modeling. In: Krause, A., Brunskill, E., Cho, K., Enge lhardt, B., Sabato, S., Scarlett, J. (eds.) International Conference o n Machine Learn- ing, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Pr...

work page 2023
[4]

In: Proceedings of the 17th Conference of the Eu ropean Chapter of the Association for Computational Linguistics

Zhou, Y., Long, G.: Multimodal event transformer for imag e-guided story ending generation. In: Proceedings of the 17th Conference of the Eu ropean Chapter of the Association for Computational Linguistics. pp. 3434–3444 (2023) 12 A. Carlos et al

work page 2023
[5]

arXiv preprin t arXiv:2410.19732 (2024)

Zhou, Y., Rao, Z., Wan, J., Shen, J.: Rethinking visual dep endency in long-context reasoning for large vision-language models. arXiv preprin t arXiv:2410.19732 (2024)

arXiv 2024
[6]

In: Findings of the Association for Comput ational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024

Zhou, Y., Li, X., Wang, Q., Shen, J.: Visual in-context lea rning for large vision- language models. In: Findings of the Association for Comput ational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024. pp. 15890– 15902. Association for Computational Linguistics (2024)

work page 2024
[7]

In: Findings of the Association for Computational Ling uistics: EACL 2023

Zhou, Y., Long, G.: Style-aware contrastive learning for multi-style image caption- ing. In: Findings of the Association for Computational Ling uistics: EACL 2023. pp. 2257–2267 (2023)

work page 2023
[8]

In: Findings of the Associ ation for Computational Linguistics: ACL 2023

Zhou, Y., Shen, T., Geng, X., Tao, C., Xu, C., Long, G., Jiao , B., Jiang, D.: Towards robust ranker for text retrieval. In: Findings of the Associ ation for Computational Linguistics: ACL 2023. pp. 5387–5401 (2023)

2023

Show all 22 references

[9]

In : Proceedings of the AAAI Conference on Artiﬁcial Intelligence

Zhou, Y., Shen, T., Geng, X., Tao, C., Shen, J., Long, G., Xu , C., Jiang, D.: Fine-grained distillation for long document retrieval. In : Proceedings of the AAAI Conference on Artiﬁcial Intelligence. vol. 38, pp. 19732–1 9740 (2024)

2024
[10]

In: International confer- ence on machine learning

Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping la nguage-image pre-training for uniﬁed vision-language understanding and generation. In: International confer- ence on machine learning. pp. 12888–12900. PMLR (2022)

2022
[11]

Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-langua ge models with instruction tuning (2023), https://arxiv.org/abs/2305.06500

2023 arXiv
[12]

In: Oh, A., Naumann, T., Glo berson, A., Saenko, K., Hardt, M., Levine, S

Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo , P., Lu, T., Zhou, J., Qiao, Y., Dai, J.: Visionllm: Large language model is als o an open-ended de- coder for vision-centric tasks. In: Oh, A., Naumann, T., Glo berson, A., Saenko, K., Hardt, M., Levine, S. (eds.) ...

2023
[13]

CoRR abs/2312.14238 (2023)

Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhon g, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., Dai, J.: I nternvl: Scal- ing up vision foundation models and aligning for generic vis ual-linguistic tasks. CoRR abs/2312.14238 (2023). https://d...

2023 doi arXiv
[14]

In: Pro- ceedings of the 2021 Conference of the North American Chapte r of the Associa- tion for Computational Linguistics: Human Language Techno logies

Zhou, Y., Geng, X., Shen, T., Zhang, W., Jiang, D.: Improv ing zero-shot cross- lingual transfer for multilingual question answering over knowledge graph. In: Pro- ceedings of the 2021 Conference of the North American Chapte r of the Associa- tion for Computational Linguistics...

2021
[15]

: Modeling event- pair relations in external knowledge graphs for script reas oning

Zhou, Y., Geng, X., Shen, T., Pei, J., Zhang, W., Jiang, D. : Modeling event- pair relations in external knowledge graphs for script reas oning. Findings of the Association for Computational Linguistics: ACL-IJCNLP 20 21 (2021)

2021
[16]

CoRR abs/2406.08394 (2024)

Wu, J., Zhong, M., Xing, S., Lai, Z., Liu, Z., Wang, W., Che n, Z., Zhu, X., Lu, L., Lu, T., Luo, P., Qiao, Y., Dai, J.: Visionllm v2: An end -to-end gen- eralist multimodal large language model for hundreds of vis ion-language tasks. CoRR abs/2406.08394 (2024). https://doi.or...

2024 doi arXiv
[17]

In: Cai, J., Kankanhalli, M.S., Prabhakaran, B., Boll, S., Subramanian, R., Zheng, L., Singh, V.K., César , P., Xie, L., Xu, D

Guo, X., Chai, W., Li, S., Wang, G.: Llava-ultra: Large ch inese language and vision assistant for ultrasound. In: Cai, J., Kankanhalli, M.S., Prabhakaran, B., Boll, S., Subramanian, R., Zheng, L., Singh, V.K., César , P., Xie, L., Xu, D. (eds.) Proceedings of the 32nd ACM Int...

2024
[18]

CoRR abs/2402.11411 (2024)

Zhou, Y., Cui, C., Rafailov, R., Finn, C., Yao, H.: Aligni ng modali- ties in vision large language models via preference ﬁne-tun ing. CoRR abs/2402.11411 (2024). https://doi.org/10.48550/ARXIV.2402.11411, https://doi.org/10.48550/arXiv.2402.11411

2024 doi arXiv
[19]

CoRR abs/2405.10292 (2024)

Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suh r, A., Xie, S., LeCun, Y., Ma, Y., Levine, S.: Fine-tuning large vision- language models as decision-making agents via reinforcement learni ng. CoRR abs/2405.10292 (2024). https://doi.org/10.48550/ARXIV.2405.10292, h...

2024 doi arXiv
[20]

CoRR abs/2408.09017 (2024)

Mombaerts, L., Ding, T., Banerjee, A., Felice, F., Taws, J., Borogo- vac, T.: Meta knowledge for retrieval augmented large langu age models. CoRR abs/2408.09017 (2024). https://doi.org/10.48550/ARXIV.2408.09017, https://doi.org/10.48550/arXiv.2408.09017

2024 doi arXiv
[21]

In: Wooldridge, M.J., Dy, J.G., Natarajan, S

Long, X., Zeng, J., Meng, F., Ma, Z., Zhang, K., Zhou, B., Z hou, J.: Generative multi-modal knowledge retrieval with large language model s. In: Wooldridge, M.J., Dy, J.G., Natarajan, S. (eds.) Thirty-Eighth AAAI Conferen ce on Artiﬁcial Intelli- gence, AAAI 2024, Thirty-Sixt...

2024
[2024]

8845–8854

pp. 8845–8854. ACM (2024). https://doi.org/10.1145/3664647.3681584, https://doi.org/10.1145/3664647.3681584

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags

Qi, D., Zhao, H., Wei, Z., Li, S.: Reminding multimodal lar ge lan- guage models of object-aware knowledge with retrieved tags . CoRR abs/2406.10839 (2024). https://doi.org/10.48550/ARXIV.2406.10839, https://doi.org/10.48550/arXiv.2406.10839

work page Pith review arXiv doi:10.48550/arxiv.2406.10839 2024

[2] [1]

In: Gold- berg, Y., Kozareva, Z., Zhang, Y

Chen, W., Hu, H., Chen, X., Verga, P., Cohen, W.W.: Murag: M ultimodal retrieval- augmented generator for open question answering over image s and text. In: Gold- berg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2 022 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2022 , Abu Dhabi, United Arab Emirates, December 7-11, 2...

doi:10.18653/v1/2022.emnlp-main.375 2022

[3] [3]

In: Krause, A., Brunskill, E., Cho, K., Enge lhardt, B., Sabato, S., Scarlett, J

Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskove c, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.: Retrieval-augmented m ultimodal lan- guage modeling. In: Krause, A., Brunskill, E., Cho, K., Enge lhardt, B., Sabato, S., Scarlett, J. (eds.) International Conference o n Machine Learn- ing, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Pr...

work page 2023

[4] [4]

In: Proceedings of the 17th Conference of the Eu ropean Chapter of the Association for Computational Linguistics

Zhou, Y., Long, G.: Multimodal event transformer for imag e-guided story ending generation. In: Proceedings of the 17th Conference of the Eu ropean Chapter of the Association for Computational Linguistics. pp. 3434–3444 (2023) 12 A. Carlos et al

work page 2023

[5] [5]

arXiv preprin t arXiv:2410.19732 (2024)

Zhou, Y., Rao, Z., Wan, J., Shen, J.: Rethinking visual dep endency in long-context reasoning for large vision-language models. arXiv preprin t arXiv:2410.19732 (2024)

arXiv 2024

[6] [6]

In: Findings of the Association for Comput ational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024

Zhou, Y., Li, X., Wang, Q., Shen, J.: Visual in-context lea rning for large vision- language models. In: Findings of the Association for Comput ational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- 16, 2024. pp. 15890– 15902. Association for Computational Linguistics (2024)

work page 2024

[7] [7]

In: Findings of the Association for Computational Ling uistics: EACL 2023

Zhou, Y., Long, G.: Style-aware contrastive learning for multi-style image caption- ing. In: Findings of the Association for Computational Ling uistics: EACL 2023. pp. 2257–2267 (2023)

work page 2023

[8] [8]

In: Findings of the Associ ation for Computational Linguistics: ACL 2023

Zhou, Y., Shen, T., Geng, X., Tao, C., Xu, C., Long, G., Jiao , B., Jiang, D.: Towards robust ranker for text retrieval. In: Findings of the Associ ation for Computational Linguistics: ACL 2023. pp. 5387–5401 (2023)

2023