Pith. sign in

REVIEW 5 major objections 5 minor 61 references

ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Add-on cuts vision-model hallucinations by up to 44 percent

desk verdict A cheap hallucination-reducing add-on with a real empirical signal, but the 'reminder composition' mechanism is not established—the implemented module is a linear bias, and the missing ablations leave the causal claim unsupported. read the letter →

arxiv 2506.22636 v1 pith:L6FNBVFJ submitted 2025-06-27 cs.CV

Sotirios Panagiotis Chytas , Miso Choi , Hyunwoo J. Kim , Vikas Singh This is my paper
classification cs.CV
keywords fadingmemoryeffecthallucinationmitigationvision-languagemodelsvectorsymbolicarchitecturesgeometricalgebradirectpreferenceoptimizationremindercompositionblack-boxintervention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models drift: after roughly forty generated tokens, the image stops influencing the predicted distribution, so the model answers from language priors alone. This paper claims that the drift, called the fading memory effect, can be countered by a small trainable reminder attached to the output layer of a frozen VLM. The reminder, ReCo, bundles the image's embeddings into one vector and adds a learned projection of it to the current token embedding before decoding, keeping the image present at every step. Trained on a 1,853-image preference dataset, ReCo cuts hallucination rates across InstructBLIP, LLaVA, and MiniGPT4 on five benchmarks and stacks with other mitigation methods.

What carries the argument

The load-bearing mechanism is the reminder composition $B_t = W_T T_t + W_I(\bigoplus_{j=1}^M I_j)$: a learnable linear recombination of the current text hidden state and a bundled sum of all image-token embeddings, inserted just before the prediction head. The bundling (sum) follows geometric-algebra and vector-symbolic-architecture ideas of composing vectors, while the linear maps keep it trainable and cheap. It works by re-injecting a pooled image 'reminder' into every next-token embedding so the image influence cannot decay during long generation.

What would settle it

Run ReCo on a benchmark of spatial-relation queries (e.g., 'is the cup to the left of the plate?' from GQA) and compare its gains with its gains on object-existence queries: if the pooled-vector reminder improves existence answers but not positional answers, the linear pooled composition is shown to be the limiting mechanism rather than a true relational composition.

Watch

Extended reading notes

Core claim

ReCo's central claim is that hallucination in VLMs is a decoding-time conditioning failure and that the fix is an explicit composition of the visual and textual state: $B_t = W_T T_t + W_I(\bigoplus_{j=1}^M I_j)$, where $T_t$ is the LLM's hidden state at step $t$ and the $I_j$ are the image embeddings. Both matrices are trainable, the VLM is frozen, and setting $W_I=0$ recovers the original model, so the modification is a strict superset of the original behavior. With only two extra lines of code, the paper reports consistent hallucination reductions for InstructBLIP, MiniGPT4, and LLaVA on POPE, CHAIR, AMBER, HallusionBench, and MME, larger gains on long generations (CHAIR$_s$ down up to 44 percent), and compatibility with existing methods such as VCD, M3ID, and AvisC, where ReCo improves each one.

Load-bearing premise

The method assumes that pooling the entire image into one summed vector and adding its learned projection to the text embedding is enough to keep the model faithful, and that the 1,853-image HA-DPO preference set is enough to train the two matrices well.

Editorial extensions

If this is right

  • Any frozen VLM can take ReCo with two lines of code and no change to its internals or inference-time cost.
  • Adding ReCo on top of existing mitigation methods (VCD, M3ID, AvisC) improves each of them on the same benchmarks.
  • Long generations benefit most: CHAIR$_s$ reductions grow with output length, up to about 44 percent for InstructBLIP/LLaVA.
  • Models that fail at yes/no answering, like MiniGPT4, gain the ability to answer such questions at a near-100 percent format rate.
  • ReCo lets a model stop elaborating on an early hallucinated object, effectively recovering from an initial mistake.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would replace the summed pool with a richer bind (tensor product or holographic reduced representation) to preserve spatial relations; the paper's framing implies this could help, but it is not tested there.
  • Because the pooling collapses location, positional and relational benchmarks may show smaller gains than object-existence ones; this is a testable corollary that the paper does not draw.
  • The same 'reminder at the head' idea could transfer to video and audio LLMs, where the conditioning signal also fades over long generation.
  • The reported gains rest on a single small preference dataset; scaling or diversifying the preference data could change the magnitude of the effect, a point the authors acknowledge as a limitation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ReCo, a lightweight trainable module placed at the output layer of a frozen vision-language model, which computes the next-token embedding as B_t = W_T T_t + W_I (⊕_j I_j), where T_t is the current hidden state and ⊕_j I_j is a pooled image embedding. ReCo is trained with the HA-DPO preference dataset (1853 images) and is claimed to mitigate the 'fading memory effect' of VLMs, thereby reducing hallucinations. Experiments are reported on InstructBLIP, MiniGPT4, and LLaVA, using CHAIR, POPE, AMBER, HallusionBench, and MME benchmarks, both standalone and in combination with three existing decoding-based mitigation methods (VCD, M3ID, AvisC). The paper reports broad improvements, with particularly large gains on long caption generation, and claims compatibility and complementary benefits with other methods.

Significance. If the central claims were fully supported, ReCo would be a valuable contribution: it is a black-box, lightweight, training-based add-on that can be deployed without modifying the VLM, and it shows sizable reductions in hallucination on several benchmarks, especially for long generations. The paper's strengths are its breadth of evaluation (multiple VLMs, multiple benchmark families) and its explicit combination with several baseline mitigation methods. However, the empirical evidence as presented is mixed: multiple cells in the paper's own tables show degradations, and the claimed mechanistic basis ('reminder composition' via geometric algebra / vector symbolic binding) is not supported by the actual linear-additive implementation. The paper does not isolate the visual term's contribution, and it provides no direct measurement of the fading memory effect before/after ReCo. These gaps make the current version of the central claim unsubstantiated, although they appear addressable with additional experiments and careful rewording.

major comments (5)
  1. [Sec. 3, Eqs. (4)-(6) and Fig. 4] The implementation in Eq. (6), B_t = W_T T_t + W_I (⊕_j I_j), is an affine map: the image term is a single vector computed once per image and added to the text hidden state at every step. This is not a bind or geometric-product operation as advertised in Eqs. (4)-(5), where the composition is written as a product T_{t-1} ⊗ (⊕_j I_j) ⊗ p_t. The reference to 'Matrix Binder' in the sentence after Eq. (6) does not make a weighted sum into a product. Consequently, the claimed GA/VSA motivation is not realized by the method, and the mechanism cannot in principle encode the spatial/relational details that Section 4.2.6 attributes to it. At minimum, the paper should rename the operation and clearly state that Eq. (6) is a pragmatic linear approximation of the intended composition, or show how the linear map can express binding.
  2. [Secs. 4.2.2-4.2.4, Tables 2, 3, 6, 7] The text repeatedly claims that ReCo 'provides a significant performance boost to all the methods' (Sec. 4.2.2), 'consistently improves the performance across all models, questions, and datasets' (Sec. 4.2.2), and that 'ReCo always improves the results' (Sec. 4.2.4). These claims are contradicted by the paper's own tables. Examples include: Table 2, InstructBLIP AvisC+ReCo on MSCoco Random (Acc -0.9%, F1 -1.2%); Table 2, MiniGPT4 AvisC+ReCo on MSCoco Random (Acc -2.1%, F1 -7.1%); Table 3, AMBER MiniGPT4 AvisC+ReCo (-2.6); Table 6, MiniGPT4 AvisC+ReCo on GQA Random (Acc -2.4%, F1 -13.5%). Also, Table 1 shows that at short generation lengths (32/64 tokens), ReCo often increases CHAIR scores (e.g., MiniGPT4 Original+ReCo: CHAIRs +2.6 at 32, +2.2 at 64; InstructBLIP Original+ReCo: CHAIRs +0.8 at 32). The paper must revise these absolute claims and instead characterize the effect as beneficial primarily at longer generations, with mixed or negative effects at short lengths and for certain combinations.
  3. [Sec. 4.1, Experiments] No ablation isolates the visual term W_I (⊕_j I_j) in Eq. (6). ReCo is trained with HA-DPO preference data while freezing the VLM, so the reported improvements could in principle be due to the DPO objective applied to the output head, rather than to the image-dependent 'reminder'. The comparison set includes no control with W_I frozen at zero (i.e., a trainable linear head with the same DPO training), no control with a constant image-independent bias, and no native HA-DPO baseline (the paper mentions HA-DPO in Sec. 4.1 and Table 4 but reports no quantitative results for it). Without these controls, the central mechanistic claim that the visual reminder is causally responsible for the gains is not established.
  4. [Sec. 1 and Sec. 4, Figure 2] The paper's title and motivation focus on the fading memory effect, and Figure 2 illustrates this effect in an unmodified VLM. However, no experiment measures fading memory before and after applying ReCo. The paper does not report the Hellinger distance between conditioned and unconditioned next-token distributions as a function of token position with and without ReCo, nor does it show attention on visual tokens after ReCo. All evidence for the mechanism is indirect, via hallucination benchmarks. A direct before/after measurement of fading memory is needed to support the paper's central explanatory claim.
  5. [Tables 1-8, throughout] The paper reports all results as point estimates without standard deviations, error bars, or significance tests. Many deltas are very small (e.g., +0.1%, +0.2% in Tables 2 and 6), and some are negative, so it is impossible to assess whether the positive claims are robust or within run-to-run noise. Given the mixed signs across the tables, reporting variance across at least a few random seeds (or using paired significance tests) is essential to support the language of 'significant performance boost'.
minor comments (5)
  1. [Abstract and throughout] The model name is spelled inconsistently: 'LlaVA' (Abstract), 'Llava' (Tables 3, 5, 6, 7, 8), 'LLaVA' (Sec. 4.2.3). Please standardize to 'LLaVA'.
  2. [Sec. 3, Eq. (6)] The symbol ⊕_j I_j is not explicitly defined. State whether it denotes mean pooling, sum pooling, or concatenation, and specify the resulting dimension so the shape of W_I is clear.
  3. [Sec. 4.2.1] The text says Table 1 shows reductions of CHAIRs 'as much as 44%', but the largest reduction visible in Table 1 is 22.2 points for MiniGPT4 AvisC+ReCo at 1024 tokens; the 44% figure appears to refer to the LLaVA results in Appendix E.2. Please clarify which table supports this claim.
  4. [Appendix C] The hyperparameter section states 'We combine ReCo with two widely used VLMs: InstructBLIP and MiniGPT4', but experiments also include LLaVA. This should be corrected to three VLMs.
  5. [Appendix D] There are several typos in the benchmark descriptions, e.g., 'disriminative', 'direct contect', and 'MSCoco'. A careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ReCo is trained on held-out preference data and evaluated on independent hallucination benchmarks, and its design is an explicit, acknowledged choice rather than a derivation from its own outputs.

full rationale

ReCo's parameters are optimized with HA-DPO preference data (Sec. 4.1) and then evaluated on POPE, CHAIR, AMBER, HallusionBench, and MME, none of which supply the training labels. The training objective and the benchmark metrics are conceptually related, but the benchmark numbers are not fitted by construction, so the reported improvements are not statistically forced. Equation (6) is introduced as a design: the paper states 'Many composition rules have been proposed in the literature but our choice above was driven by two reasons' and 'our formulation depends on a specific composition strategy,' so the geometric-algebra and VSA preamble is motivational rather than a derivation whose output equals its input. The one self-citation, reference [27], is used only to support the general statement that composition is relevant to interpretability and semantic validity; it is not load-bearing, and no uniqueness theorem is imported from the authors' prior work. Appendix B's caveat that the GPT-4-generated chosen/rejected pairs 'may not be entirely accurate' is a data-quality limitation, not a circular step. The absence of an ablation isolating W_I and the lack of a post-training re-measurement of the fading-memory diagnostic are experimental gaps, not reductions of the central claim to its inputs. Overall, no claimed prediction is equivalent to an input by definition or by the paper's own equations.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; the 'multi-vector' is implemented as a linear combination of existing embeddings. The main assumptions are about the sufficiency of pooled image features and the HA-DPO data.

free parameters (6)
  • W_T = learned during DPO
    Linear transform on text hidden state in Eq. (6); trained on HA-DPO preference data.
  • W_I = learned during DPO
    Linear projection of pooled image embeddings in Eq. (6); trained on HA-DPO preference data.
  • learning_rate = 5e-3
    Chosen by hand (Appendix C); affects convergence of the module.
  • beta = 0.8
    HA-DPO hyperparameter (Appendix C), chosen by hand.
  • lambda = 0.2
    HA-DPO hyperparameter (Appendix C), chosen by hand.
  • epochs = 10
    Training duration (Appendix C), chosen by hand.
assumptions (4)
  • domain assumption The fading memory effect is the main cause of VLM hallucination.
    Intro and Fig. 2, based on prior work (e.g., [10]); the paper does not prove this causality, but uses it to justify the design.
  • domain assumption All image information relevant for generation can be captured by a single pooled vector ⊕_j I_j.
    Eq. (6) collapses M image tokens into one vector before composing with text; spatial and relational structure is lost.
  • domain assumption HA-DPO preference data generated by GPT-4 is a sufficient training signal.
    Appendix B notes that C and R prompts 'may not be entirely accurate' but assumes DPO still works.
  • domain assumption The VLM head input can be modified by an additive image term without changing the model's learned text distribution in harmful ways.
    Sec. 3 claims W_T = I and W_I = 0 restores the original, but no analysis is provided for other values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Models." pith.science (2026). https://pith.science/paper/L6FNBVFJ

@misc{pith2026250622636,
  author       = {Pith},
  title        = {Pith review of: ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6FNBVFJ}},
  note         = {Machine review of arXiv:2506.22636}
}
read the original abstract

Vision Language Models (VLMs) show impressive capabilities in integrating and reasoning with both visual and language data. But these models make mistakes. A common finding -- similar to LLMs -- is their tendency to hallucinate, i.e., generate plausible sounding text which is not grounded in the visual input, or at worst, is contradictory. A growing consensus attributes this behavior to an over-reliance on language -- especially as the generation progresses, the model suffers from a ``fading memory effect'' with respect to the provided visual input. We study mechanisms by which this behavior can be controlled. Specifically, using ideas from geometric algebra and relational compositions, we propose the addition of a small, trainable module (named ReCo) on top of any VLM -- no other modification is needed. We show that such a lightweight module is able to mitigate the fading memory effect on three of the most widely used VLMs (InstructBLIP, LlaVA, MiniGPT4), where we see performance improvements on multiple benchmarks. Additionally, we show that our module can be combined with many of the other approaches for reducing hallucination where we achieve improved results for each one.

Figures

Figures reproduced from arXiv: 2506.22636 by the authors.

Figure 1
Figure 1. InstructBLIP [1] before and after ReCo. We propose a small module that, with minimal training, it is able to effectively re￾duce the hallucination rate of widely used VLMs. Given the advances in the capabilities of Large Lan￾guage Models (LLMs), recent efforts have sought to extend these models to the multi-modality setting, i.e., processing and “understanding” additional modali￾ties beyond text such as audio, image… view at source ↗
Figure 2
Figure 2. The “fading memory effect”. Top: First-layer attention of the visual and textual input as the generation progresses. Bottom: The effect of the visual input on the log￾its distribution as the generation progresses. We calculate the next token prediction with and without the visual input and we com￾pute the distributional difference. After the first tokens, the next token can be predicted just from the previously gene… view at source ↗
Figure 3
Figure 3. VLM: ideal versus practice. On the left, we show what actually happens, where at each timestep the image’s influence (𝐼) in or￾ange is diminished compared to the text (𝑇𝑡 ) in blue. So, the generated text is not an accurate representation of the visual input. An ideal VLM (right side) would form an object (𝐵𝑡 ) that perfectly encapsulates all of the given in￾put, leading to accurate generation. The desired behavior.… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: ReCo overview. The VLM is treated as a black box, modifying the next token embedding by combining the multi-vector of visual tokens and the current token prediction. First, we bundle the image tokens [𝐼𝑗 ] 𝑚 𝑗=1 into a single vector 𝐼 and, then, bind it with 𝑇𝑡 to form…
Figure 5
Figure 5. Figure 5: ReCo in practice. No access to the LLM is required and the only change is the modification of the prediction head with the addition of a “preprocessing” step. VLM is a black-box. Notice that (6) and the correspond￾ing code (fig. 5) involves only the output layer of the…
Figure 6
Figure 6. Figure 6: POPE [32] results on MiniGPT4 [6]. The unmodified version is often unable to comprehend the question and outputs an unrelated answer that does no contain either “Yes” or “No”. On the contrary, ReCo provides the model the ability to answer such questions [PITH_FULL_IMA…
Figure 7
Figure 7. Figure 7: MiniGPT4 out of the box as well as other methods are not able to answer properly (with a Yes or No) discrimina￾tive questions in most cases. In contrast, ReCo offers the model the ability to comprehend/an￾swer such questions. As we see in fig. 6, MiniGPT4 is not able t…
Figure 8
Figure 8. Figure 8: Structural hallucinations: MiniGPT4 [6] before (red) and after ReCo (green). The unmodi￾fied VLM tends to get small details about the scene wrong, like the texture of the floor or the written signs that are depicted in the image. Enabling ReCo fixes such mistakes. 8 […
Figure 9
Figure 9. Figure 9: Failure of all MiniGPT4-based models but ReCo to answer the AMBER [34] questions coherently. All models describe the image (with some of them getting the de￾tails wrong) although the question is a binary (yes/no) one (whose true label is “Yes”). CHAIR [33] and POPE [32…
Figure 10
Figure 10. Figure 10: A sample of the HA-DPO dataset [14]. While the 𝐶 and 𝑅 prompts may not be entirely accurate since they were generated with the help of LLMs [14], the Direct Preference Optimization [13] successfully trains ReCo. C Hyperparameters We combine ReCo with two widely used V…
Figure 11
Figure 11. Figure 11: An image obtain from HallusionBench [35]. The corresponding question is: “According [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Generative questions on InstructBLIP [1]. The prompt used is [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Generative questions on MiniGPT4 [6]. The prompt used is [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Discriminative questions on MiniGPT4 [6], taken from the AMBER benchmarks. The unmodified VLM fails to answer cohesively in most of the cases, while the addition of ReCo (although not trained on such data) allows the model to answer correctly. 20 [PITH_FULL_IMAGE:fig…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 38 canonical work pages

  1. [1]

    InstructBLIP: Towards general-purpose vision-language modelswithinstruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language modelswithinstruction tuning. In Thirty-seventh Conference onNeural InformationProcessing Systems, 2023. URLhttps://openreview.net/forum?id=vvoWPYqZJA

    work page 2023

  2. [2]

    Visual instruction tun- ing

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tun- ing. InAdvances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023. URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ 6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf

    work page 2023

  3. [3]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    arXiv 2023

  4. [4]

    AudioChatLlama: To- wards general-purpose speech abilities for LLMs

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. AudioChatLlama: To- wards general-purpose speech abilities for LLMs. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    work page doi:10.18653/v1/2024.naacl-long.309 2024

  5. [5]

    Video-llama: Aninstruction-tunedaudio-visuallanguage modelforvideounderstanding

    HangZhang,XinLi,andLidongBing. Video-llama: Aninstruction-tunedaudio-visuallanguage modelforvideounderstanding. arXivpreprintarXiv:2306.02858 ,2023. URL https://arxiv. org/abs/2306.02858

    arXiv 2023

  6. [6]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    arXiv 2023

  7. [7]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023. URL https://arxiv.org/abs/2310.09478

    arXiv 2023

  8. [8]

    Improved baselines with visual instructiontuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instructiontuning. InProceedingsoftheIEEE/CVFConferenceonComputerVisionandPattern Recognition (CVPR), 2024

    work page 2024

Show all 61 references
  1. [9]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, NikolayBashlykov,SoumyaBatra,PrajjwalBhargava,ShrutiBhosale,DanBikel,LukasBlecher, CristianCantonFerrer,MoyaChen,GuillemCucurull,DavidEsiobu,JudeFernandes,JeremyFu, Wenyin Fu, Brian Fuller,...

    2023 arXiv

  2. [10]

    Multi-modal hallucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In2024 IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  3. [11]

    Mitigatingobjecthallucinationsinlargevision-languagemodelsthroughvisualcontrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigatingobjecthallucinationsinlargevision-languagemodelsthroughvisualcontrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2024

  4. [12]

    Don’t miss the forest for the trees: Attentional vision calibration for large vision language models.arXiv preprint arXiv:2405.17820, 2024

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don’t miss the forest for the trees: Attentional vision calibration for large vision language models.arXiv preprint arXiv:2405.17820, 2024

    2024 arXiv

  5. [13]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openre...

    2023

  6. [14]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization, 2023

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization, 2023

    2023

  7. [15]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual in- struction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual in- struction data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  8. [16]

    Hallucination augmented contrastive learning for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  9. [17]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Conference...

    2024

  10. [18]

    Reducing hallucinations in vision-language models via latent space steering, 2024

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. Reducing hallucinations in vision-language models via latent space steering, 2024. URLhttps://arxiv.org/abs/2410.15778

    2024 arXiv

  11. [19]

    Attributes as operators: Factorizing unseen attribute- object compositions

    Tushar Nagarajan and Kristen Grauman. Attributes as operators: Factorizing unseen attribute- object compositions. InComputer Vision – ECCV 2018, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01246-5

    2018

  12. [20]

    Dávila, Alfredo Gómez-Rodríguez, and Marco Rodríguez-Andrade

    Gerardo Aragón-González, José Aragón, F. Dávila, Alfredo Gómez-Rodríguez, and Marco Rodríguez-Andrade. Modern Geometric Calculations in Crystallography. 2001. ISBN 0-8176- 4199-8. doi: 10.1007/978-1-4612-0159-5_18

    2001 doi

  13. [21]

    Geometric algebra, 2012

    Eric Chisolm. Geometric algebra, 2012. URLhttps://arxiv.org/abs/1205.5935

    2012 arXiv

  14. [22]

    Learning graph embeddings for composi- tionalzero-shotlearning

    MF Naeem, Y Xian, F Tombari, and Zeynep Akata. Learning graph embeddings for composi- tionalzero-shotlearning. In 34thIEEEConferenceonComputerVisionandPatternRecognition . IEEE, 2021

    2021

  15. [23]

    Open world compositional zero-shot learning

    Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi: 10.1109/CVPR46437.2021.00518

    2021

  16. [24]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In...

    2017 doi

  17. [25]

    From red wine to red tomato: Composition with context

    Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

  18. [26]

    Neural-symbolic vqa: disentangling reasoning from vision and language understanding

    KexinYi,JiajunWu,ChuangGan,AntonioTorralba,PushmeetKohli,andJoshuaB.Tenenbaum. Neural-symbolic vqa: disentangling reasoning from vision and language understanding. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, Red Hook, ...

    2018

  19. [27]

    Kim, and Vikas Singh

    Sotirios Panagiotis Chytas, Hyunwoo J. Kim, and Vikas Singh. Understanding multi- compositional learning in vision and language models via category theory. InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, 2024, 2024

    2024

  20. [28]

    Learningwithholographicreducedrepresentations

    AshwinkumarGanesan,HangGao,SunilGandhi,EdwardRaff,TimOates,JamesHolt,andMark McLean. Learningwithholographicreducedrepresentations. In AdvancesinNeuralInformation Processing Systems, 2021. URLhttps://openreview.net/forum?id=RX6PrcpXP-

    2021

  21. [29]

    Gunn and Steven De Keninck

    Charles G. Gunn and Steven De Keninck. Geometric algebra and computer graphics. In ACM SIGGRAPH 2019 Courses, SIGGRAPH ’19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450363075. doi: 10.1145/3305366.3328099. URL https://doi.org/10.1145/3305366.3328099

    2019

  22. [30]

    Geometric algebra trans- former

    Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra trans- former. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=M7r2CO4tJC

    2023

  23. [31]

    Gupta, Steven De Keninck, Max Welling, and Johannes Brandstetter

    David Ruhe, Jayesh K. Gupta, Steven De Keninck, Max Welling, and Johannes Brandstetter. Geometric clifford algebra networks. InProceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  24. [32]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URLhttps://openreview.net/forum? id=xozJw0kZXF

    2023

  25. [33]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. InEmpirical Methods in Natural Language Processing (EMNLP), 2018

    2018

  26. [34]

    An llm-free multi-dimensional benchmark for mllms hallucination evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023

    2023 arXiv

  27. [35]

    Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  28. [36]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  29. [37]

    Relational composition in neural networks: A survey and call to action

    Martin Wattenberg and Fernanda Viégas. Relational composition in neural networks: A survey and call to action. InICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=zzCEiUIPk9

    2024

  30. [38]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. URLhttps://arxiv. org/pdf/1706.03762.pdf

    2017 arXiv

  31. [39]

    Pooling methods in deep neural networks, a review, 2020

    Hossein Gholamalinezhad and Hossein Khosravi. Pooling methods in deep neural networks, a review, 2020. URLhttps://arxiv.org/abs/2009.07485

    2020 arXiv

  32. [40]

    Look, ma, no matrices! InACM SIGGRAPH 2024 Talks, SIGGRAPH ’24, New York, NY, USA, 2024

    Steven De Keninck. Look, ma, no matrices! InACM SIGGRAPH 2024 Talks, SIGGRAPH ’24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705151. doi: 10.1145/3641233.3665801. URL https://doi.org/10.1145/3641233.3665801

    2024

  33. [41]

    T.A. Plate. Holographic reduced representations.IEEE Transactions on Neural Networks, 6(3),

  34. [42]

    Vector-derivedtransformationbinding: Animprovedbinding operation for deep symbol-like processing in neural networks.Neural Comput., 31(5), 2019

    JanGosmannandChrisEliasmith. Vector-derivedtransformationbinding: Animprovedbinding operation for deep symbol-like processing in neural networks.Neural Comput., 31(5), 2019. ISSN 0899-7667. doi: 10.1162/neco_a_01179. URLhttps://doi.org/10.1162/neco_a_ 01179

    2019 doi

  35. [43]

    Thomason

    Richmond H. Thomason. Logic and artificial intelligence. InThe development of modern logic. Oxford University Press, 2009

    2009

  36. [44]

    Fodor and Zenon W

    Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A crit- ical analysis. Cognition, 28(1), 1988. ISSN 0010-0277. doi: https://doi.org/10.1016/ 0010-0277(88)90031-5. URL https://www.sciencedirect.com/science/article/ pii/0010027788900315

    1988

  37. [45]

    Tensor product variable binding and the representation of symbolic structures in connectionist systems.Artificial Intelligence, 46(1), 1990

    Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems.Artificial Intelligence, 46(1), 1990. ISSN 0004-3702. doi: https://doi. org/10.1016/0004-3702(90)90007-M. URL https://www.sciencedirect.com/science/ article/p...

    1990

  38. [46]

    Touretzky and Geoffrey E

    David S. Touretzky and Geoffrey E. Hinton. Symbols among the neurons: Details of a connec- tionist inference architecture. InInternational Joint Conference on Artificial Intelligence, 1985. URL https://api.semanticscholar.org/CorpusID:14138392

    1985

  39. [47]

    Acomparisonofvectorsymbolicarchitectures

    KennySchlegel,PeerNeubert,andPeterProtzel. Acomparisonofvectorsymbolicarchitectures. Artif. Intell. Rev., 55(6), 2022. ISSN 0269-2821. doi: 10.1007/s10462-021-10110-3. URL https://doi.org/10.1007/s10462-021-10110-3

    2022 doi

  40. [48]

    URLhttps://api.semanticscholar.org/ CorpusID:52182870

    MatthiasWolff,GüntherWirsching,MarkusHuber,PeterbeimGraben,RonaldRömer,andIngo Schmitt.Afockspacetoolboxandsomeapplicationsincomputationalcognition.In International Conference on Speech and Computer, 2018. URLhttps://api.semanticscholar.org/ CorpusID:52182870

    2018

  41. [49]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV2014 ,Cham,2014.SpringerInternationalPublishing. ISBN978-3-319-10602-1

    2014

  42. [50]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision – ECCV 2022, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 20074-8

    2022

  43. [51]

    Gqa: Anewdatasetforreal-worldvisualreasoning andcompositionalquestionanswering

    DrewAHudsonandChristopherDManning. Gqa: Anewdatasetforreal-worldvisualreasoning andcompositionalquestionanswering. In ProceedingsoftheIEEE/CVFconferenceoncomputer vision and pattern recognition, pages 6700–6709, 2019

    2019

  44. [52]

    A survey on hallucination in large vision-language models, 2024

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models, 2024. URL https://arxiv.org/abs/2402.00253

    2024 arXiv

  45. [53]

    Recasting self-attention with holographic reduced representations

    Mohammad Mahmudul Alam, Edward Raff, Stella Biderman, Tim Oates, and James Holt. Recasting self-attention with holographic reduced representations. InProceedings of the 40th InternationalConferenceonMachineLearning ,volume202of ProceedingsofMachineLearning Research. PMLR, 2023...

    2023

  46. [56]

    C Hyperparameters We combine ReCo with two widely used VLMs: InstructBLIP [1] and MiniGPT4 [6] and we follow the same training procedure for both

    successfully trains ReCo. C Hyperparameters We combine ReCo with two widely used VLMs: InstructBLIP [1] and MiniGPT4 [6] and we follow the same training procedure for both. The learning rate is set to5𝑒-3and we train the model for10 epochs. Additionally, the𝛽 and𝜆parameters of...

  47. [57]

    Istherea ⟨object⟩intheimage?

    POPE [32]: The POPE benchmark is focused on binary existence questions. It consists of 9000questionsintotal,oftheform “Istherea ⟨object⟩intheimage?” ,eachoneaccompanied by an image. POPE is designed to measure the disriminative capabilities of the underlying VLM, in the contex...

  48. [58]

    Describe the image

    CHAIR [33]: While POPE is focused on the discriminative capabilities of the VLMs, CHAIRassessestheirgenerativepower. Thisbenchmarkconsistsofimagesaccompaniedby theirgroundtruthlabels,i.e.,whichobjectsexistineachoneofthem. TheVLMisprompted with a prompt such as“Describe the ima...

  49. [59]

    Is there a direct contect between⟨object1⟩and⟨object2⟩

    AMBER [34]: AMBER combines and extends CHAIR and POPE, by evaluating the under- lyingVLMonbothawidearrayofdiscriminativequestionsaboutobjectsrelationships(e.g., “Is there a direct contect between⟨object1⟩and⟨object2⟩”), objects attributes (e.g.,“Is the cloud black in this imag...

  50. [60]

    Similarly to AMBER, MME also consists of multiple types of discriminative questions such asartworkrelated, coderelated,and numericalrelatedquestions

    MME[36]: MME is yet another, larger, discriminative benchmark. Similarly to AMBER, MME also consists of multiple types of discriminative questions such asartworkrelated, coderelated,and numericalrelatedquestions. Foreachimage,MMEproposestwoquestions (one whose correct answer i...

  51. [61]

    According to the image, in 2017, was Tencent the company with the highest revenue from video games, with Sony as the second-highest earner?

    HallusionBench[35]: SimilartoMME,HallusionBenchintroducesawidearrayofdifferent discrimination questions. Differently than the other benchmarks, most of the questions in thisbenchmarkareaboutcharts,tables,andmapsdepictedonthegivenimages. Anexample of a image-question pair can b...

    2017

  52. [1995]

    doi: 10.1109/72.377968. 12

    doi

  53. [2017]

    doi: 10.1109/CVPR.2017.129

    2017 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.