{"id":"e9266d6b-c155-4928-bc7e-ccd0c89266f9","arxiv_id":"2411.13053","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multimodal explanation-guided learning framework that jointly uses visual saliency maps and textual rationales to train image classifiers, improving accuracy, visual explanation overlap, and text explanation scores on two new datasets.","lead":"MEGL is a training method that uses both visual heatmaps and written rationales as extra supervision for image classifiers, improving accuracy and explanation quality. It is a step toward more transparent AI, showing that combining two explanation types beats using either alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage risk in the distribution-consistency target A_bar: if A_bar aggregates test-fold annotations, reported mIoU and accuracy gains are inflated; the split is unspecified.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the distribution-consistency loss supervises the majority of images without visual annotations using an aggregated target, but the paper never specifies the data split, A_bar normalization, or whether test annotations contribute to A_bar. For the paper's central comparative claim, this is the most direct threat because visual mIoU is measured against ground-truth saliency maps; if those same maps were used to build A_bar, the evaluation is circular. Even in the absence of explicit leakage, the aggregated target assumes a single dataset-level saliency pattern is a meaningful supervisory signal for all unannotated images, which is not justified by any analysis or ablations. The paper's own admission that textual metrics may be inflated by templated explanations further weakens the textual side of the claim, but the visual leakage risk is more consequential because it could invalidate both the mIoU and the classification gains if the training pipeline inadvertently mixes test annotations. My stress test does not move the verdict: CONDITIONAL remains appropriate, with the condition being a precise statement of the data split, A_bar construction, and error bars before the comparative results can be taken at face value.","tokens_in":14547,"tokens_out":2766,"duration_ms":28709,"concrete_test":"Obtain the released code/data (or ask the authors for the exact split). Recompute A_bar twice: (i) using only train-fold annotations and (ii) using all annotations. Retrain MEGL-ViT-B/16 from the paper's settings and report accuracy and mIoU on a held-out test split with 95% confidence intervals. If mIoU under (i) drops materially relative to Table 2, or if (i) and (ii) differ substantially, the reported visual-explanation gains are leakage-inflated. As a second check, replace Ldc with a fixed random map for the unannotated images; if accuracy/mIoU changes insignificantly, the consistency loss is not performing the claimed supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is comparative: MEGL outperforms prior EGL and MLLM baselines in accuracy, visual mIoU, and textual quality. For that claim to hold, evaluation must be on held-out data and the training signal must not incorporate test information. Section 3.4.2 defines A_bar as the average of normalized ground-truth saliency maps over 'the annotated samples' without stating whether these are training-only. Section 4.1 reports only dataset totals (402/1,185 annotated samples), and Section 4.4 omits train/validation/test splits. If A_bar is computed over all annotated samples including those later used for mIoU, then the visual-explanation loss for unannotated training samples is supervised by test annotations, and the mIoU in Table 2 measures overlap against the very target used in training. Even without explicit test leakage, using one global average map as the target for ~90% of images (Object-ME: 402/4,790; Action-ME: 1,185/11,511) is a strong, unvalidated modeling assumption; the Ldc term then provides a generic prior rather than image-specific grounding. The reported margins are also unquantified: MEGL-ViT-B/16 beats RES-L on Object-ME mIoU by 0.3521 vs 0.3401, with no error bars or significance tests anywhere. This is load-bearing: the headline 'outperforms all SOTA' rests on unspecified data partitioning and an unvalidated aggregated target.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MEGL, a framework for image classification that trains a classifier with both visual saliency supervision and textual rationale supervision. The method uses Saliency-Driven Textual Grounding (SDTG) to feed saliency-weighted image features into an LLM, and a Visual Explanation Distribution Consistency loss for images lacking visual annotations. The authors construct two new datasets, Object-ME and Action-ME, derived from VQA-X and ACT-X, and report experiments comparing MEGL against vision models, visual EGL baselines (CDEP, HAICS, RES-G, RES-L), and MLLM/EGL baselines (LLaVA, Fine-tune-CoT). The central claim is that MEGL outperforms all previous methods in classification accuracy, visual explanation mIoU, and textual explanation quality.","tokens_in":14830,"tokens_out":4311,"duration_ms":37788,"significance":"If the empirical claims are validated, MEGL is a useful contribution to explanation-guided learning, demonstrating a practical way to integrate multimodal explanations and to handle partially annotated modalities. The paper also contributes two new benchmark datasets and an efficiency analysis. However, the current evidence is weakened by the absence of a described data split, the ambiguous and potentially leaky construction of the distribution-consistency target, and the lack of uncertainty quantification. These issues must be resolved before the comparative claims can be accepted; the central idea is sound and fixable within the manuscript's scope.","major_comments":[{"comment":"The aggregated target A_bar is defined as an average over normalized ground-truth saliency maps without specifying whether these maps come only from training samples. Since Section 4.4 never describes a train/validation/test split, it is possible that A_bar includes annotations from samples that are later used to compute mIoU in Table 2. If so, the Ldc term trains unannotated samples against test annotations and the reported mIoU gains (e.g., MEGL-ViT-B/16 0.3521 vs RES-L 0.3401 on Object-ME) are inflated by leakage. Please specify the split, restrict A_bar to the training fold, and report mIoU only on a held-out set.","section":"Section 3.4.2"},{"comment":"The visual distribution-consistency loss is defined inconsistently: in Section 3.2 it is Ldc = E[||A_hat - A||] (an average L1 distance over the set of ground-truth maps), while in Section 3.4.2 it is Ldc = D_KL(A_hat || A_bar) where A_bar is the aggregated map. These are different objectives and lead to different training behavior. Please state the exact loss used in the final objective (equation in Section 3.2) and in the ablation study (Table 4), and fix the notation (Lds vs Ldc).","section":"Section 3.2 vs 3.4.2"},{"comment":"No error bars, confidence intervals, significance tests, or multiple-seed results are reported. The headline comparison 'MEGL-ViT-B/16 outperforms all SOTA baseline models' rests on small margins (e.g., Object-ME mIoU 0.3521 vs 0.3401; Object-ME accuracy 0.8317 vs 0.8206 for RES-L). Without run-to-run variance, these differences cannot be distinguished from noise. Please add standard deviations over at least three seeds, or equivalent significance testing, for the main metrics.","section":"Tables 2 and 3"},{"comment":"Implementation details omit several components needed to reproduce the experiments: the size and construction of the train/validation/test splits, the source of the 402 (Object-ME) and 1,185 (Action-ME) visual annotations (training-only or partitioned), the values of lambda_visual and lambda_textual, and the number of fine-tuning epochs and random seeds. These details are necessary to assess whether the reported mIoU is computed on held-out samples and whether the distribution-consistency loss is computed without target leakage.","section":"Section 4.4"},{"comment":"The paper acknowledges that 'the high scores achieved across various language metrics may be attributed to the relatively templated nature of textual explanations in our datasets.' This caveat directly affects the textual-explainability comparison, since templated outputs can inflate n-gram metrics such as BLEU/ROUGE/CIDEr. Please provide an analysis of explanation diversity or a human evaluation to support the claim of superior textual explanation quality; CLIPScore alone does not establish faithfulness.","section":"Section 4.5"}],"minor_comments":[{"comment":"There is a typo in 'Visual Question Answering Explanation (VQA-X) and Activity Explanation (ACT-X)' where 'tecxtual' should be 'textual'.","section":"Section 4.1"},{"comment":"In the sentence 'MEGL-ViT-B/16 also generates texutal explanations of higher quality', 'texutal' should be 'textual'.","section":"Section 4.5"},{"comment":"The model name 'MEGL-ViT-16/B' in Table 5 should be 'MEGL-ViT-B/16' for consistency with the rest of the paper.","section":"Table 5"},{"comment":"The symbol 'Lds' appears once in the sentence 'With the proposed consistency loss Lds'; it should be 'Ldc' to match the rest of the section.","section":"Section 3.4.2"},{"comment":"In the introduction, 'Similiarly' should be 'Similarly'.","section":"Introduction"},{"comment":"The word 'evalauations' in 'we conduct comprehensive evalauations' should be 'evaluations'.","section":"Section 4.3"},{"comment":"The caption contains a duplicated 'to to' in 'reported to to compare classification performance'.","section":"Table 2 caption"},{"comment":"Reference [14] lists the author as 'Zhao Liang'; this should likely be 'Liang Zhao' to match the other references to this author.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising and the datasets could be useful to the community, but the experimental protocol requires substantial clarification. I recommend requesting the code/data release and a clear specification of the split and evaluation protocol. The comparison against LLaVA-Fine-tune-CoT is also confounded by model size; the efficiency table helps but does not fully address the accuracy comparison. I would not reject on these grounds if the authors can provide the missing details and uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: this paper proposes a sensible next step in explanation-guided learning—combine visual and textual explanation supervision in one training loop—and the main mechanism (SDTG, feeding a Grad-CAM-masked image embedding into an LLM) is plausible. But the headline claim that MEGL beats all SOTA rests on an underspecified evaluation setup, and until the authors say what the data split is and give error bars, I wouldn't take the numbers at face value.\n\nWhat's actually new: prior EGL work is single-modality. MEGL combines both, and the SDTG architecture is a reasonable way to ground textual rationales in saliency. The distribution-consistency loss for unannotated images is an interesting workaround for annotation scarcity. The two derived datasets (Object-ME, Action-ME) could be useful for future benchmarking. The paper also honestly flags that text metrics are likely inflated by template explanations; that's a good sign.\n\nThe soft spots are real. The biggest is the data split. Section 4.4 describes implementation details but never says how many samples go to train/val/test. That matters because the distribution-consistency target A_bar is a dataset-level average of ground-truth saliency maps. If A_bar is computed on all annotated samples including those used for mIoU evaluation, then the model is trained against the test target and the visual explanation metric is partly self-scoring. The text says 'the annotated samples' without restricting to training. I don't know if this is actual leakage or just sloppy writing, but the paper must say which. The stress-test note makes the same point and I think it's the load-bearing question.\n\nAlso, there are no error bars, multiple seeds, or significance tests anywhere. The reported gains are modest—mIoU 0.3521 vs 0.3401 on Object-ME with ViT-B/16—so the comparative claim could easily be noise. The loss notation is inconsistent: Lvisual is L1 in Eq. 1, but Ldc is KL in Section 3.4.2, yet the combined objective writes them under one LVisual. That's fixable, but it makes the method harder to pin down. No code or data are released, which makes all of this worse.\n\nWho is this for? People working on XAI/EGL who want a concrete multimodal extension. If the authors can clarify the split, add seeds/error bars, and release code, this could be a useful contribution. As it stands, it deserves a serious referee—the idea is good—but it needs major revision before I'd trust the numbers.\n\nI'd send it to review, but I wouldn't build on it yet.\n\nBest,","headline":"A plausible multimodal extension of EGL that is currently undermined by an unspecified data split and missing error bars; worth one serious revision, not a desk reject.","tokens_in":15360,"tokens_out":2440,"would_cite":false,"duration_ms":23008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEGL shows that jointly supervising an image classifier with visual saliency maps and textual rationales improves accuracy, visual explanation overlap, and textual explanation quality, beating existing explanation-guided learning…","keywords":["multimodal explanation-guided learning","image classification","visual explanations","textual rationales","saliency grounding","distribution consistency loss","Object-ME","Action-ME"],"falsifier":"Re-run the comparison with a strict split of the visual-annotation subset so that the dataset-average saliency target is computed exclusively from training images, and recompute mIoU only on test images whose ground-truth maps were excluded from that average; if the advantage over the baselines shrinks or vanishes, the reported visual gains came from the target rather than from multimodal alignment.","tokens_in":14309,"feed_emoji":"🧠","tokens_out":11305,"duration_ms":94766,"temperature":0.7,"pith_summary":"This paper proposes MEGL, a framework that trains an image classifier under joint supervision from both visual saliency maps and textual rationales. The authors construct two datasets, Object-ME and Action-ME, in which every image has a class label and a human-written textual explanation, while only a small subset has visual explanation annotations. They report that MEGL exceeds existing explanation-guided learning baselines and multimodal large language models on classification accuracy, visual explanation overlap, and textual explanation quality on both datasets, and that it does so with much lower inference cost than LLM-based classifiers. The central claim is that making the two explanation modalities supervise each other during training simultaneously sharpens predictions and makes the explanations more faithful.","feed_headline":"Paired visual and textual supervision lifts image classifier accuracy","feed_subtitle":"MEGL jointly supervises classifiers with saliency maps and textual rationales, beating EGL and MLLM baselines on two new datasets.","key_machinery":"The central object is the combined training objective that ties the three supervision signals together. The named component is Saliency-Driven Textual Grounding (SDTG), which injects the classifier's saliency-weighted image into an LLM along with the full image features, so the generated rationale is forced to respect spatial cues. The objective is $L = L_{\\text{pred}} + \\lambda_{\\text{textual}} L_{\\text{textual}} + \\lambda_{\\text{visual}}(I_V L_{\\text{visual}} + (1-I_V) L_{dc})$, where $L_{\\text{visual}}$ is an L1 loss between the Grad-CAM map and human annotation, $L_{dc}$ is a KL divergence between the saliency map and a dataset-averaged saliency target, and the indicator $I_V$ switches between them depending on whether the image has a visual annotation.","core_discovery":"The paper's central discovery is that visual and textual explanations can be made to reinforce each other during training, and that this joint supervision improves both the classifier and the explanations it produces. The method, Saliency-Driven Textual Grounding (SDTG), takes the saliency map produced by Grad-CAM on the classifier, multiplies it into the image, encodes that saliency-weighted image with a CLIP encoder, and feeds the resulting token sequence together with the original image features into a language model that generates the textual rationale. That rationale is trained with an autoregressive loss against human-written text, and gradients propagate back to the visual explainer, providing what the paper calls Textual Supervision on Visual Explanations. For images without ground-truth saliency maps, the paper introduces a Visual Explanation Distribution Consistency loss that pulls the generated saliency map toward a dataset-level average of the annotated maps. On the new Object-ME and Action-ME datasets, the paper reports that MEGL with a ViT-B/16 backbone achieves the best classification, visual mIoU, and textual explanation scores among all compared baselines.","pith_inferences":["Beyond the paper, the dataset-level average target could likely be sharpened into a class-conditional average, which would give the weak-supervision loss more precision when different classes have different visual patterns.","The same saliency-into-text injection could be run in reverse, using only textual rationales to supervise the visual explainer, which would extend the approach to datasets that have language explanations but no saliency annotations at all.","Because the paper's own ablation notes the measured text quality may be inflated by templated rationales, a further test would apply MEGL to open-ended explanation datasets to see whether the multimodal grounding gains persist with more varied language."],"forward_implications":["On both constructed datasets, MEGL with a ViT-B/16 backbone beats all compared visual-EGL, textual-EGL, and MLLM baselines on accuracy, macro-averaged precision/recall/F1, and visual mIoU.","The textual-supervision path makes visual explanation learning possible even when only a small fraction of images carry saliency annotations, so partially annotated collections become usable for explanation-guided training.","The distribution-consistency loss allows the remaining unannotated images to still shape the saliency training signal, which is the mechanism behind the reported visual explainability gains.","As a deployed classifier, MEGL is far lighter than LLM-based alternatives (about 85M parameters versus 7B) and shows a 30x FPS speedup while maintaining higher measured accuracy."],"supporting_citations":[{"why":"Supplies the VQA-X and ACT-X source datasets from which Object-ME and Action-ME are constructed.","marker":"[40]"},{"why":"Provides the Grad-CAM visual explainer that generates the saliency maps supervised in the framework.","marker":"[47]"},{"why":"Provides the CLIP vision encoder used to encode the saliency-weighted image in the textual grounding path.","marker":"[43]"},{"why":"Provides the Vicuna v1.5 language model used to generate the textual rationales.","marker":"[61]"},{"why":"Serves as the LLaVA MLLM baseline and the checkpoint from which MEGL fine-tunes.","marker":"[34]"},{"why":"Provides the Fine-tune-CoT baseline for textual explanation-guided learning.","marker":"[23]"},{"why":"Provides the CDEP visual EGL baseline that supervises explanations alongside predictions.","marker":"[45]"},{"why":"Provides the RES-G and RES-L visual explanation guidance baselines.","marker":"[14]"}],"fun_headline_variants":["Visual and textual explanations reinforce each other to train better classifiers","MEGL turns explanations into supervision, boosting accuracy and interpretability","Using visual and textual explanations as training signal lifts image classification","Saliency maps and text rationales jointly supervise classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single dataset-level average of the annotated saliency maps is a meaningful and leakage-free teaching target for the majority of images that lack their own visual annotations, so the reported gains depend on that average being built only from training images and never from the images used to compute mIoU.","fun_headline_variants_meta":{"raw":{"variants":["Visual and textual explanations reinforce each other to train better classifiers","MEGL turns explanations into supervision, boosting accuracy and interpretability","Using visual and textual explanations as training signal lifts image classification","Saliency maps and text rationales jointly supervise classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2937,"prompt_tokens":1030,"completion_tokens":1907,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":1839}},"tokens_in":646,"tokens_out":1907,"duration_ms":15086,"temperature":1.0,"reasoning_tokens":1839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:52:20.857988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with a strict split of the visual-annotation subset so that the dataset-average saliency target is computed exclusively from training images, and recompute mIoU only on test images whose ground-truth maps were excluded from that average; if the advantage over the baselines shrinks or vanishes, the reported visual gains came from the target rather than from multimodal alignment.","supporting_citations":[{"cited_title":"Multimodal explanations: Justifying deci- sions and pointing to the evidence","cited_arxiv_id":null,"evidence_quote":"Supplies the VQA-X and ACT-X source datasets from which Object-ME and Action-ME are constructed."},{"cited_title":"10 Grad-cam: Visual explanations from deep networks via gradient-based localization","cited_arxiv_id":null,"evidence_quote":"Provides the Grad-CAM visual explainer that generates the saliency maps supervised in the framework."},{"cited_title":"Interpretations are useful: penalizing explanations to align neural networks with prior knowledge","cited_arxiv_id":null,"evidence_quote":"Provides the CDEP visual EGL baseline that supervises explanations alongside predictions."},{"cited_title":"Res: A robust framework for guiding visual explanation","cited_arxiv_id":null,"evidence_quote":"Provides the RES-G and RES-L visual explanation guidance baselines."}],"review_version":1}