Pith. sign in

REVIEW 5 major objections 8 minor 1 cited by

MEGL: Multimodal Explanation-Guided Learning

T0 review · 5 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MEGL shows that jointly supervising an image classifier with visual saliency maps and textual rationales improves accuracy, visual explanation overlap, and textual explanation quality, beating existing explanation-guided learning…

desk verdict A plausible multimodal extension of EGL that is currently undermined by an unspecified data split and missing error bars; worth one serious revision, not a desk reject. read the letter →

arxiv 2411.13053 v1 pith:PVCEO75J submitted 2024-11-20 cs.CV cs.AIcs.LG

Yifei Zhang , Tianxu Jiang , Bo Pan , Jingyu Wang , Guangji Bai , Liang Zhao This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords multimodalexplanation-guidedlearningimageclassificationvisualexplanationstextualrationalessaliencygroundingdistributionconsistencylossObject-MEAction-ME
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes MEGL, a framework that trains an image classifier under joint supervision from both visual saliency maps and textual rationales. The authors construct two datasets, Object-ME and Action-ME, in which every image has a class label and a human-written textual explanation, while only a small subset has visual explanation annotations. They report that MEGL exceeds existing explanation-guided learning baselines and multimodal large language models on classification accuracy, visual explanation overlap, and textual explanation quality on both datasets, and that it does so with much lower inference cost than LLM-based classifiers. The central claim is that making the two explanation modalities supervise each other during training simultaneously sharpens predictions and makes the explanations more faithful.

What carries the argument

The central object is the combined training objective that ties the three supervision signals together. The named component is Saliency-Driven Textual Grounding (SDTG), which injects the classifier's saliency-weighted image into an LLM along with the full image features, so the generated rationale is forced to respect spatial cues. The objective is $L = L_{\text{pred}} + \lambda_{\text{textual}} L_{\text{textual}} + \lambda_{\text{visual}}(I_V L_{\text{visual}} + (1-I_V) L_{dc})$, where $L_{\text{visual}}$ is an L1 loss between the Grad-CAM map and human annotation, $L_{dc}$ is a KL divergence between the saliency map and a dataset-averaged saliency target, and the indicator $I_V$ switches between them depending on whether the image has a visual annotation.

What would settle it

Re-run the comparison with a strict split of the visual-annotation subset so that the dataset-average saliency target is computed exclusively from training images, and recompute mIoU only on test images whose ground-truth maps were excluded from that average; if the advantage over the baselines shrinks or vanishes, the reported visual gains came from the target rather than from multimodal alignment.

Watch

Extended reading notes

Core claim

The paper's central discovery is that visual and textual explanations can be made to reinforce each other during training, and that this joint supervision improves both the classifier and the explanations it produces. The method, Saliency-Driven Textual Grounding (SDTG), takes the saliency map produced by Grad-CAM on the classifier, multiplies it into the image, encodes that saliency-weighted image with a CLIP encoder, and feeds the resulting token sequence together with the original image features into a language model that generates the textual rationale. That rationale is trained with an autoregressive loss against human-written text, and gradients propagate back to the visual explainer, providing what the paper calls Textual Supervision on Visual Explanations. For images without ground-truth saliency maps, the paper introduces a Visual Explanation Distribution Consistency loss that pulls the generated saliency map toward a dataset-level average of the annotated maps. On the new Object-ME and Action-ME datasets, the paper reports that MEGL with a ViT-B/16 backbone achieves the best classification, visual mIoU, and textual explanation scores among all compared baselines.

Load-bearing premise

The load-bearing premise is that a single dataset-level average of the annotated saliency maps is a meaningful and leakage-free teaching target for the majority of images that lack their own visual annotations, so the reported gains depend on that average being built only from training images and never from the images used to compute mIoU.

Editorial extensions

If this is right

  • On both constructed datasets, MEGL with a ViT-B/16 backbone beats all compared visual-EGL, textual-EGL, and MLLM baselines on accuracy, macro-averaged precision/recall/F1, and visual mIoU.
  • The textual-supervision path makes visual explanation learning possible even when only a small fraction of images carry saliency annotations, so partially annotated collections become usable for explanation-guided training.
  • The distribution-consistency loss allows the remaining unannotated images to still shape the saliency training signal, which is the mechanism behind the reported visual explainability gains.
  • As a deployed classifier, MEGL is far lighter than LLM-based alternatives (about 85M parameters versus 7B) and shows a 30x FPS speedup while maintaining higher measured accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the dataset-level average target could likely be sharpened into a class-conditional average, which would give the weak-supervision loss more precision when different classes have different visual patterns.
  • The same saliency-into-text injection could be run in reverse, using only textual rationales to supervise the visual explainer, which would extend the approach to datasets that have language explanations but no saliency annotations at all.
  • Because the paper's own ablation notes the measured text quality may be inflated by templated rationales, a further test would apply MEGL to open-ended explanation datasets to see whether the multimodal grounding gains persist with more varied language.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces MEGL, a framework for image classification that trains a classifier with both visual saliency supervision and textual rationale supervision. The method uses Saliency-Driven Textual Grounding (SDTG) to feed saliency-weighted image features into an LLM, and a Visual Explanation Distribution Consistency loss for images lacking visual annotations. The authors construct two new datasets, Object-ME and Action-ME, derived from VQA-X and ACT-X, and report experiments comparing MEGL against vision models, visual EGL baselines (CDEP, HAICS, RES-G, RES-L), and MLLM/EGL baselines (LLaVA, Fine-tune-CoT). The central claim is that MEGL outperforms all previous methods in classification accuracy, visual explanation mIoU, and textual explanation quality.

Significance. If the empirical claims are validated, MEGL is a useful contribution to explanation-guided learning, demonstrating a practical way to integrate multimodal explanations and to handle partially annotated modalities. The paper also contributes two new benchmark datasets and an efficiency analysis. However, the current evidence is weakened by the absence of a described data split, the ambiguous and potentially leaky construction of the distribution-consistency target, and the lack of uncertainty quantification. These issues must be resolved before the comparative claims can be accepted; the central idea is sound and fixable within the manuscript's scope.

major comments (5)
  1. [Section 3.4.2] The aggregated target A_bar is defined as an average over normalized ground-truth saliency maps without specifying whether these maps come only from training samples. Since Section 4.4 never describes a train/validation/test split, it is possible that A_bar includes annotations from samples that are later used to compute mIoU in Table 2. If so, the Ldc term trains unannotated samples against test annotations and the reported mIoU gains (e.g., MEGL-ViT-B/16 0.3521 vs RES-L 0.3401 on Object-ME) are inflated by leakage. Please specify the split, restrict A_bar to the training fold, and report mIoU only on a held-out set.
  2. [Section 3.2 vs 3.4.2] The visual distribution-consistency loss is defined inconsistently: in Section 3.2 it is Ldc = E[||A_hat - A||] (an average L1 distance over the set of ground-truth maps), while in Section 3.4.2 it is Ldc = D_KL(A_hat || A_bar) where A_bar is the aggregated map. These are different objectives and lead to different training behavior. Please state the exact loss used in the final objective (equation in Section 3.2) and in the ablation study (Table 4), and fix the notation (Lds vs Ldc).
  3. [Tables 2 and 3] No error bars, confidence intervals, significance tests, or multiple-seed results are reported. The headline comparison 'MEGL-ViT-B/16 outperforms all SOTA baseline models' rests on small margins (e.g., Object-ME mIoU 0.3521 vs 0.3401; Object-ME accuracy 0.8317 vs 0.8206 for RES-L). Without run-to-run variance, these differences cannot be distinguished from noise. Please add standard deviations over at least three seeds, or equivalent significance testing, for the main metrics.
  4. [Section 4.4] Implementation details omit several components needed to reproduce the experiments: the size and construction of the train/validation/test splits, the source of the 402 (Object-ME) and 1,185 (Action-ME) visual annotations (training-only or partitioned), the values of lambda_visual and lambda_textual, and the number of fine-tuning epochs and random seeds. These details are necessary to assess whether the reported mIoU is computed on held-out samples and whether the distribution-consistency loss is computed without target leakage.
  5. [Section 4.5] The paper acknowledges that 'the high scores achieved across various language metrics may be attributed to the relatively templated nature of textual explanations in our datasets.' This caveat directly affects the textual-explainability comparison, since templated outputs can inflate n-gram metrics such as BLEU/ROUGE/CIDEr. Please provide an analysis of explanation diversity or a human evaluation to support the claim of superior textual explanation quality; CLIPScore alone does not establish faithfulness.
minor comments (8)
  1. [Section 4.1] There is a typo in 'Visual Question Answering Explanation (VQA-X) and Activity Explanation (ACT-X)' where 'tecxtual' should be 'textual'.
  2. [Section 4.5] In the sentence 'MEGL-ViT-B/16 also generates texutal explanations of higher quality', 'texutal' should be 'textual'.
  3. [Table 5] The model name 'MEGL-ViT-16/B' in Table 5 should be 'MEGL-ViT-B/16' for consistency with the rest of the paper.
  4. [Section 3.4.2] The symbol 'Lds' appears once in the sentence 'With the proposed consistency loss Lds'; it should be 'Ldc' to match the rest of the section.
  5. [Introduction] In the introduction, 'Similiarly' should be 'Similarly'.
  6. [Section 4.3] The word 'evalauations' in 'we conduct comprehensive evalauations' should be 'evaluations'.
  7. [Table 2 caption] The caption contains a duplicated 'to to' in 'reported to to compare classification performance'.
  8. [References] Reference [14] lists the author as 'Zhao Liang'; this should likely be 'Liang Zhao' to match the other references to this author.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MEGL's losses are training objectives that supervise a classifier, not predictions reconstructed from the reported metrics, and no load-bearing result is imported from same-author citations.

full rationale

The paper's claimed derivation chain is an empirical training procedure rather than a mathematical derivation. The final objective L = Lpred + lambda_textual Ltextual + lambda_visual (IV Lvisual + (1 - IV) Ldc) directly optimizes classification, textual likelihood, and saliency alignment; the reported mIoU is a separate measured overlap metric, not the same expression as the L1 visual loss or the KL distribution-consistency loss, so no reported number is identical to a training term by construction. The distribution-consistency target A_bar is an aggregate of annotated ground-truth saliency maps, and whether that annotated set includes test-fold samples is an unspecified data-split and potential leakage concern, not a demonstrated circularity: the paper never defines the evaluation mIoU in terms of A_bar or defines A_bar in terms of the held-out test labels. Self-citations to RES, ESSA, MAGI, and the EGL survey appear in related work and as comparison baselines, but the MEGL formulation does not rest on any uniqueness theorem, ansatz, or unverified prior result from the same authors; the baselines are externally published methods with fixed reported behavior. The paper even candidly notes in Section 4.5 that high language metric scores 'may be attributed to the relatively templated nature of textual explanations,' acknowledging a confound rather than concealing it. No step in Sections 3.2-3.4 reduces a claimed prediction to a fitted parameter or to the paper's own outputs by definition, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on two tunable hyperparameters, the faithfulness of Grad-CAM as an explanation method, the quality of the source VQA-X and ACT-X explanations, and the validity of using a dataset-level average saliency map as a supervision target. The most fragile item is the unstated split and aggregation protocol, which directly affects the trustworthiness of all reported metrics.

free parameters (2)
  • lambda_textual
    Hyperparameter weighting the textual explanation loss in the final objective. No value is reported in the paper, and it is presumably tuned on validation data.
  • lambda_visual
    Hyperparameter weighting the visual explanation loss and the distribution consistency loss. No value is reported.
assumptions (4)
  • domain assumption Grad-CAM saliency maps are a faithful proxy for the classifier's decision-relevant regions.
    The visual explanation A_hat is generated by Grad-CAM and used both as supervision and as input to the textual grounding path (Sections 3.2 and 3.3). If Grad-CAM highlights irrelevant regions, the entire multimodal alignment is built on poor signal.
  • domain assumption The ground-truth visual and textual explanations in Object-ME and Action-ME are accurate and complete.
    These explanations are taken from VQA-X and ACT-X without audit. The textual rationales are used as targets for L_textual and the visual annotations as targets for L_visual, so noise or hallucination in the source datasets would propagate into training.
  • ad hoc to paper The dataset-level average of normalized saliency maps is a valid training target for unannotated samples.
    Section 3.4.2 introduces Ldc to align generated saliency maps with A_bar, the average of all ground-truth saliency maps. This assumes a shared spatial pattern across a dataset, which is not generally true for image classification and is not empirically validated.
  • ad hoc to paper The evaluation is performed on a held-out split and A_bar is computed only from training annotations.
    The paper never specifies the data split or how A_bar is computed relative to train/test. This assumption is required for the reported mIoU and accuracy numbers to be unbiased. Without it, the results could reflect leakage or training-set memorization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MEGL: Multimodal Explanation-Guided Learning." pith.science (2026). https://pith.science/paper/PVCEO75J

@misc{pith2026241113053,
  author       = {Pith},
  title        = {Pith review of: MEGL: Multimodal Explanation-Guided Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PVCEO75J}},
  note         = {Machine review of arXiv:2411.13053}
}
read the original abstract

Explaining the decision-making processes of Artificial Intelligence (AI) models is crucial for addressing their "black box" nature, particularly in tasks like image classification. Traditional eXplainable AI (XAI) methods typically rely on unimodal explanations, either visual or textual, each with inherent limitations. Visual explanations highlight key regions but often lack rationale, while textual explanations provide context without spatial grounding. Further, both explanation types can be inconsistent or incomplete, limiting their reliability. To address these challenges, we propose a novel Multimodal Explanation-Guided Learning (MEGL) framework that leverages both visual and textual explanations to enhance model interpretability and improve classification performance. Our Saliency-Driven Textual Grounding (SDTG) approach integrates spatial information from visual explanations into textual rationales, providing spatially grounded and contextually rich explanations. Additionally, we introduce Textual Supervision on Visual Explanations to align visual explanations with textual rationales, even in cases where ground truth visual annotations are missing. A Visual Explanation Distribution Consistency loss further reinforces visual coherence by aligning the generated visual explanations with dataset-level patterns, enabling the model to effectively learn from incomplete multimodal supervision. We validate MEGL on two new datasets, Object-ME and Action-ME, for image classification with multimodal explanations. Experimental results demonstrate that MEGL outperforms previous approaches in prediction accuracy and explanation quality across both visual and textual domains. Our code will be made available upon the acceptance of the paper.

Figures

Figures reproduced from arXiv: 2411.13053 by the authors.

Figure 1
Figure 1. Comparison of visual and textual explanations for an im [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the MEGL Framework. The framework is jointly trained to optimize prediction accuracy, visual explainability, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints for Reliable Decision-Making

    cs.CV 2026-01 unverdicted novelty 5.0 of 10

    Penalizing off-prior attribution evidence during training with subset-selection attribution improves accuracy and attribution reasonability in image classifiers and MLLM-based GUI agents.

Reference graph

Works this paper leans on

62 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Tbexplain: A text-based explanation method for scene classification models with the statistical prediction correction

    Amirhossein Aminimehr, Pouya Khani, Amirali Molaei, Amirmohammad Kazemeini, and Erik Cambria. Tbexplain: A text-based explanation method for scene classification models with the statistical prediction correction. In Proceed- ings of the Conference on Governance, Understanding and Integration of Data for Effective and Responsible AI , pages 54–60, 2024. 1, 2, 3, 4

    work page 2024

  3. [3]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Part V 14 , pages 382–398. Springer, 2016. 6

    2016

  4. [4]

    Gemini: A family of highly capable multimodal models

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1, 2023. 3

    arXiv 2023

  5. [5]

    Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai

    Alejandro Barredo Arrieta, Natalia D ´ıaz-Rodr´ıguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garc´ıa, Sergio Gil-L ´opez, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82–115, 2020. 1

    work page 2020

  6. [6]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 6

    2005

  7. [7]

    Let there be a clock on the beach: Reducing object halluci- nation in image captioning

    Ali Furkan Biten, Llu ´ıs G ´omez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object halluci- nation in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1381–1390, 2022. 2

    2022

  8. [8]

    A survey on xai and nat- ural language explanations

    Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Navid Nobani. A survey on xai and nat- ural language explanations. Information Processing & Man- agement, 60(1):103111, 2023. 1, 2, 4

    work page 2023

Show all 62 references
  1. [9]

    Machine learning interpretability: A survey on methods and metrics

    Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):832, 2019. 1

    2019

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3, 6

    2010 arXiv

  3. [11]

    Techniques for in- terpretable machine learning

    Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for in- terpretable machine learning. Communications of the ACM, 63(1):68–77, 2019. 1

    2019

  4. [12]

    Learn- ing credible deep neural networks with rationale regulariza- tion

    Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. Learn- ing credible deep neural networks with rationale regulariza- tion. In 2019 IEEE International Conference on Data Mining (ICDM), pages 150–159. IEEE, 2019. 3

    2019

  5. [13]

    Attention branch network: Learning of attention mechanism for visual explanation

    Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10705–10714, 2019. 3

    2019

  6. [14]

    Res: A robust framework for guiding visual explanation

    Yuyang Gao, Tong Steven Sun, Guangji Bai, Siyi Gu, Sung- soo Ray Hong, and Zhao Liang. Res: A robust framework for guiding visual explanation. In proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 432–442, 2022. 6, 7

    2022

  7. [15]

    Going beyond xai: A system- atic survey for explanation-guided learning

    Yuyang Gao, Siyi Gu, Junji Jiang, Sungsoo Ray Hong, Dazhou Yu, and Liang Zhao. Going beyond xai: A system- atic survey for explanation-guided learning. ACM Comput- ing Surveys, 56(7):1–39, 2024. 2

    2024

  8. [16]

    Don’t trust your eyes: on the (un) reliability of feature visualizations

    Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don’t trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.04719, 2023. 2

    2023 arXiv

  9. [17]

    Essa: Explanation iterative supervision via saliency-guided data augmentation

    Siyi Gu, Yifei Zhang, Yuyang Gao, Xiaofeng Yang, and Liang Zhao. Essa: Explanation iterative supervision via saliency-guided data augmentation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 567–576, 2023. 2, 3

    2023

  10. [18]

    Xai- class: Explanation-enhanced text classification with ex- tremely weak supervision

    Daniel Hajialigol, Hanwen Liu, and Xuan Wang. Xai- class: Explanation-enhanced text classification with ex- tremely weak supervision. arXiv preprint arXiv:2311.00189,

    arXiv

  11. [19]

    Generating faith- ful and salient text from multimodal data

    Tahsina Hashem, Weiqing Wang, Derry Tanti Wijaya, Mo- hammed Eunus Ali, and Yuan-Fang Li. Generating faith- ful and salient text from multimodal data. arXiv preprint arXiv:2409.03961, 2024. 6

    2024 arXiv

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6 9

    2016

  13. [21]

    Generating vi- sual explanations

    Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating vi- sual explanations. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oc- tober 11–14, 2016, Proceedings, Part IV 14 , pages 3–1...

    2016

  14. [22]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  15. [23]

    Large language models are reasoning teachers

    Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. 6, 7

    2022 arXiv

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 7

    2021 arXiv

  17. [25]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2256– 2264, 2024. 3

    2024

  18. [26]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. 2

    2023

  19. [27]

    Hallucination augmented contrastive learn- ing for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  20. [28]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015. 3

    2015

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  22. [30]

    Symbolic chain-of-thought distillation: Small models can also” think” step-by-step

    Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of-thought distillation: Small models can also” think” step-by-step. arXiv preprint arXiv:2306.14050, 2023. 2, 3

    2023 arXiv

  23. [31]

    Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions

    Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In Proceedings of the European Con- ference on Computer Vision (ECCV), pages 552–567, 2018. 3

    2018

  24. [32]

    Explanations from large language models make small reasoners better

    Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726 ,

    arXiv

  25. [33]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 6

    2004

  26. [34]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3, 6, 7

    2024

  27. [35]

    A unified approach to interpreting model predictions

    Scott Lundberg. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017. 1

    2017 arXiv

  28. [36]

    Teach- ing small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teach- ing small language models to reason. arXiv preprint arXiv:2212.08410, 2022. 2, 3

    2022 arXiv

  29. [37]

    Natural language rationales with full-stack visual reasoning: From pixels to semantic frames to commonsense graphs

    Ana Marasovi ´c, Chandra Bhagavatula, Jae Sung Park, Ro- nan Le Bras, Noah A Smith, and Yejin Choi. Natural language rationales with full-stack visual reasoning: From pixels to semantic frames to commonsense graphs. arXiv preprint arXiv:2010.07526, 2020. 3

    2010 arXiv

  30. [38]

    Vale: A mul- timodal visual and language explanation framework for im- age classifiers using explainable ai and language models

    Purushothaman Natarajan and Athira Nambiar. Vale: A mul- timodal visual and language explanation framework for im- age classifiers using explainable ai and language models. arXiv preprint arXiv:2408.12808, 2024. 3

    2024 arXiv

  31. [39]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318,

  32. [40]

    Multimodal explanations: Justifying deci- sions and pointing to the evidence

    Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Mar- cus Rohrbach. Multimodal explanations: Justifying deci- sions and pointing to the evidence. In Proceedings of the IEEE conference on computer vision and pattern recogni- ti...

    2018

  33. [41]

    Ro- bust explanations for visual question answering

    Badri Patro, Shivansh Patel, and Vinay Namboodiri. Ro- bust explanations for visual question answering. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1577–1586, 2020. 3

    2020

  34. [42]

    Rise: Randomized input sampling for explana- tion of black-box models

    V Petsiuk. Rise: Randomized input sampling for explana- tion of black-box models. arXiv preprint arXiv:1806.07421,

    arXiv

  35. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  36. [44]

    A first look: Towards explainable textvqa models via visual and textual explanations

    Varun Nagaraj Rao, Xingjian Zhen, Karen Hovsepian, and Mingwei Shen. A first look: Towards explainable textvqa models via visual and textual explanations. arXiv preprint arXiv:2105.02626, 2021. 3

    2021 arXiv

  37. [45]

    Interpretations are useful: penalizing explanations to align neural networks with prior knowledge

    Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. InInternational conference on machine learning, pages 8116–8126. PMLR,

  38. [46]

    Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450, 2016

    Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Ba- tra. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450, 2016. 1

    2016 arXiv

  39. [47]

    10 Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 10 Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE in- ternational conference on computer vision , pages 618–626,

  40. [48]

    Human-ai interactive and continuous sensemaking: A case study of image classification using scribble attention maps

    Haifeng Shen, Kewen Liao, Zhibin Liao, Job Doornberg, Maoying Qiao, Anton Van Den Hengel, and Johan W Ver- jans. Human-ai interactive and continuous sensemaking: A case study of image classification using scribble attention maps. In extended abstracts of the 2021 CHI conferenc...

    2021

  41. [49]

    A review of taxonomies of explainable ar- tificial intelligence (xai) methods

    Timo Speith. A review of taxonomies of explainable ar- tificial intelligence (xai) methods. In Proceedings of the 2022 ACM conference on fairness, accountability, and trans- parency, pages 2239–2250, 2022. 1

    2022

  42. [50]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. InInternational conference on machine learning, pages 3319–3328. PMLR, 2017. 3

    2017

  43. [51]

    Robustness may be at odds with accuracy

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152,

    arXiv

  44. [52]

    Explainable artificial intel- ligence (xai) in deep learning-based medical image analysis

    Bas HM Van der Velden, Hugo J Kuijf, Kenneth GA Gilhuijs, and Max A Viergever. Explainable artificial intel- ligence (xai) in deep learning-based medical image analysis. Medical Image Analysis, 79:102470, 2022. 1

    2022

  45. [53]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 6

    2015

  46. [54]

    Faithful multimodal explanation for visual question answering

    Jialin Wu and Raymond J Mooney. Faithful multimodal explanation for visual question answering. arXiv preprint arXiv:1809.02805, 2018. 3

    2018 arXiv

  47. [55]

    Generating deep networks explanations with ro- bust attribution alignment

    Guohang Zeng, Yousef Kowsar, Sarah Erfani, and James Bailey. Generating deep networks explanations with ro- bust attribution alignment. In Asian Conference on Machine Learning, pages 753–768. PMLR, 2021. 2, 3

    2021

  48. [56]

    Overlooked trustworthiness of saliency maps

    Jiajin Zhang, Hanqing Chao, Giridhar Dasegowda, Ge Wang, Mannudeep K Kalra, and Pingkun Yan. Overlooked trustworthiness of saliency maps. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pages 451–461. Springer, 2022. 2

    2022

  49. [57]

    Rationale- augmented convolutional neural networks for text classifica- tion

    Ye Zhang, Iain Marshall, and Byron C Wallace. Rationale- augmented convolutional neural networks for text classifica- tion. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing. Conference on Empiri- cal Methods in Natural Language Processing,...

    2016

  50. [58]

    Magi: Multi-annotated explanation-guided learning

    Yifei Zhang, Siyi Gu, Yuyang Gao, Bo Pan, Xiaofeng Yang, and Liang Zhao. Magi: Multi-annotated explanation-guided learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1977–1987, 2023. 2, 3

    1977

  51. [59]

    Multimodal chain-of- thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 3

    2023 arXiv

  52. [60]

    Self-explain: Teaching large language models to reason complex questions by themselves

    Jiachen Zhao, Zonghai Yao, Zhichao Yang, and Hong Yu. Self-explain: Teaching large language models to reason complex questions by themselves. arXiv preprint arXiv:2311.06985, 2023. 2, 3

    2023 arXiv

  53. [61]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6

    2023

  54. [62]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.