{"id":"ac90e3a5-e20a-47c0-9877-74d61ad1ea36","arxiv_id":"2501.15321","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"M3H, which fuses GPT-4o commonsense reasoning about memes with retrieved similar examples, improves mental health meme symptom classification over 20 baseline variations on two datasets.","lead":"This paper introduces AxiOM, a new dataset of 3,582 memes labeled with six anxiety symptoms from the GAD questionnaire, and a framework called M3H that uses GPT-4o-generated figurative reasoning plus retrieval to classify mental health memes. On AxiOM and the public RESTORE depression dataset, M3H reports weighted-F1 gains of about 4 to 5 points over prior state-of-the-art baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG may act as a label-retrieval shortcut: retrieved in-context examples carry gold labels and near-duplicate memes across train/test could inflate reported F1 gains; deduplication and retrieval-precision checks are needed.","rationale":"The reader's weakest_assumption targets GPT-4o reasoning faithfulness. I agree that hallucinated reasoning would hurt, but the paper's own examples and human evaluation suggest the reasoning is usually on-target; if anything, the reasoning so explicitly names the symptom that it enables a retrieval shortcut. The more decisive threat to the central claim is therefore the evaluation protocol: the RAG supplies labeled exemplars selected by an embedding that is built from the same reasoning that already contains the answer. If near-duplicate memes straddle the train/test split, the top-n retrieved set is a near-label-leak. The paper's inconsistencies (e.g., Table 2 ablation rows not matching Table 4, and §5.2 text mixing AxiOM/RESTORE scores) already justify a conditional verdict; the near-duplicate/retrieval-leakage check is a concrete, decisive test that could move the verdict to REJECT if the gains collapse. I recommend keeping the CONDITIONAL verdict with an explicit requirement for the deduplication and retrieval-precision analysis. This is not an accusation of fraud; it is a standard robustness check for RAG-based approaches on meme data, where re-posts are common. If the gains survive deduplication and random-retrieval replacement, the central claim is much stronger.","tokens_in":17749,"tokens_out":11606,"duration_ms":98440,"concrete_test":"Compute perceptual image hashes (e.g., pHash or dHash) for all AxiOM and RESTORE images and mark any test meme whose hash is within a small Hamming distance of a training image (or whose OCR text overlaps beyond a threshold) as a near-duplicate. Remove all near-duplicate test memes and re-run the full M3H pipeline and the strongest baseline (OCR + GPT-4o reasoning with BART) on the reduced test set. If the M3H-minus-baseline F1 gap shrinks by more than half or vanishes, the reported gains are driven by retrieval of near-duplicates rather than by commonsense reasoning. Additionally, in the same run, replace the retrieved examples in the prompt with randomly sampled training examples of equal count; if F1 collapses to the no-RAG level, the RAG gain is attributable to the retrieval mechanism rather than the content of the reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol around the RAG module, not (only) the faithfulness of GPT-4o reasoning. In the M3H prompt (Table 5), the retrieved training examples are presented with their gold labels (e.g., 'Difficulty Relaxing'). These examples are chosen by cosine similarity in the joint OCR-text + figurative-reasoning embedding space (Eq. 1-2). Since the figurative reasoning for each meme is prompted to name the cause-effect, figurative meaning, and mental state, it frequently lexically identifies the symptom class (e.g., 'Mental State: Anxiety'). Consequently, the embedding of a test meme is likely to be nearest to training memes of the same class, and the classifier can achieve high F1 by effectively copying the label of the majority retrieved example. The paper does not report retrieval precision, does not compare against random or label-flipped retrieved examples, and does not analyze near-duplicate images/text between train and test. Meme collections from the same subreddits often contain re-posts and template variants, so a test meme may have a near-identical training image, making the retrieval a direct label leak. If this shortcut explains the 4.20%/4.66% weighted-F1 deltas, the central claim that figurative-commonsense infusion drives the improvement is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AxiOM, a new dataset of 3,582 memes annotated for six GAD-derived anxiety symptoms, and proposes M3H, a framework that uses GPT-4o to generate figurative-commonsense reasoning (cause-effect, figurative understanding, mental state), retrieves similar training examples via a RAG module, and classifies with a BART head. M3H is benchmarked against 20 baseline variations on AxiOM and on the RESTORE depression dataset, with reported weighted-F1 improvements of 4.20% and 4.66% over the prior SOTA (Yadav et al.). The paper also includes ablation studies, a human evaluation of reasoning quality, and error analyses.","tokens_in":17994,"tokens_out":7044,"duration_ms":52777,"significance":"If the central claim holds, this is a useful contribution to multimodal mental-health meme classification: it provides a new clinically grounded anxiety-meme dataset, demonstrates a concrete way to inject LLM-generated figurative reasoning, and reports improvements over a wide set of baselines, with code and data released. The human evaluation and public dataset are assets. However, the specific numerical claims are weakened by internal arithmetic and ablation inconsistencies, and the RAG-based evaluation does not yet rule out a label-copying shortcut, so the paper's central attribution of gains to commonsense infusion is not yet established.","major_comments":[{"comment":"The reported improvements contain arithmetic errors that contradict the paper's own table. In Table 2, the M3H minus SOTA deltas are listed as 4.94, 4.20, 5.79, 4.66 for RESTORE macro-F1, RESTORE weighted-F1, AxiOM macro-F1, AxiOM weighted-F1. But the same table gives RESTORE macro-F1 67.52 vs 63.58 (difference 3.94, not 4.94) and AxiOM weighted-F1 70.10 vs 65.34 (difference 4.76, not 4.66). The abstract and Section 1 also state these deltas in an order that does not match Table 2 (the abstract says '4.20% and 4.66% on weighted-F1' without naming datasets; Section 1 says '4.94% and 5.79% on macro-F1 ... across AxiOM and RESTORE, respectively', while Table 2 shows the opposite mapping). Please correct these numbers and ensure the abstract, introduction, and Table 2 are mutually consistent.","section":"Abstract; Section 1; Table 2"},{"comment":"The ablation reported in Table 2 does not match the detailed ablation in Appendix Table 4. The row labeled '– (FCS+RAG)' in Table 2 lists AxiOM macro-F1 64.28 and weighted-F1 64.97, but the corresponding row in Table 4 (OCR + FCS + FCS-RAG, without OCR-RAG) lists AxiOM macro-F1 64.94 and weighted-F1 64.95. Similarly, Table 2's '– (OCR+RAG)' row lists AxiOM 62.54/62.53, while Table 4's OCR + OCR-RAG + FCS row (without FCS-RAG) lists 61.82/61.79. These discrepancies affect the paper's summary of which component contributes what. Please reconcile Table 2 with Table 4 and clarify the notation: '– (FCS+RAG)' and '– (OCR+RAG)' are ambiguous about which components are removed.","section":"Section 5.2; Appendix C, Table 4"},{"comment":"The RAG evaluation does not rule out a label-copying shortcut. In the M3H prompt (Table 5), each retrieved training example is presented with its gold label (e.g., 'Difficulty Relaxing'). Retrieval is performed by cosine similarity in the joint OCR-text + figurative-reasoning embedding space (Eqs. 1–2), and the generated figurative reasoning frequently names the mental state explicitly (e.g., 'Mental State: Anxiety'). A test meme whose reasoning mentions its true class will likely retrieve training memes of the same class, so the BART classifier can achieve high F1 by copying the majority retrieved label. The paper does not report retrieval precision@k, does not compare against random, label-flipped, or label-masked retrieved examples, and does not analyze near-duplicate images or template variants between train and test. Please add these controls; without them the reported 4.20%/4.66% weighted-F1 gains cannot be attributed to figurative-commonsense infusion rather than to label leakage through retrieval.","section":"Section 4.2; Table 5"},{"comment":"The faithfulness of GPT-4o's figurative reasoning is under-evaluated. The paper's only direct check is a human evaluation of 40 samples (Section 5.4) that rates reasoning quality on a Likert scale; it does not test whether the generated reasoning contains incorrect mental-state attributions that would mislead classification. The Discussion (Section 6) itself acknowledges that LLMs 'may produce responses misaligned with human reasoning or fail to capture meme-specific nuances'. Given that this reasoning is the core input to both the RAG database and the classifier prompt, please provide a correctness-oriented evaluation (e.g., human verdicts on whether the generated mental state matches the gold label) or an ablation using oracle, random, or corrupted reasoning to show that classification is robust to reasoning errors.","section":"Section 5.4; Section 6"},{"comment":"There are additional internal inconsistencies in the results narrative. Section 5.2 states that M3H achieves 'F1 scores of 67.52 (+494%) and 68.79 (+4.20%), and 70.09' without saying these are RESTORE's macro, weighted, and micro F1; the '+494%' is presumably '+4.94%', which is itself incorrect (see above). The same paragraph says 'we observe a consistent improvement on RESTORE data as well, with 70.00% score on both macro and weighted-F1', but Table 2 shows 70.00 for AxiOM, not RESTORE. Appendix C reports 'ΔRAG = 5.40' while the main text says '+5.09%'. Please correct these statements and make the dataset-order conventions consistent throughout.","section":"Section 5.2; Appendix C"}],"minor_comments":[{"comment":"The model name is inconsistently written as GPT-4o in most places but GPT-4 in Section 5.4, Section 7, and Figure 6; please use a single convention.","section":"Throughout"},{"comment":"The table header writes 'LLA V A' and 'OCR + LLA V A' with an unusual space; this appears to be a typesetting artifact that should be fixed to LLaVA.","section":"Table 2"},{"comment":"The notation for hyperparameters and thresholds is sometimes inconsistent: Appendix B says the learning rate scheduler was 'constant', but Table 3 lists 'constant' as the value; more importantly, the RESTORE multilabel thresholds τ_j (Eq. 3) are mentioned but their values or selection procedure are never described. Please state how τ_j were set.","section":"Section 5.1; Appendix B"},{"comment":"Figure 7 reports ΔRAG = 5.40% but Section 5.2 says '+5.09%'; please verify which number is correct and cite the figure consistently.","section":"Figure 7; Section 5.2"},{"comment":"The human evaluation uses only 40 samples (20 per dataset) and reports only the majority rating distribution; please provide inter-annotator agreement or a more detailed breakdown, and state whether the same GPT-4o-generated reasoning was used for both the RAG database and the human evaluation.","section":"Section 5.4"},{"comment":"Equation (1) uses a concatenation of embeddings to form R^{n×2d}, but the subsequent retrieval in Eq. (2) uses cosine similarity; concatenated embeddings are known to distort cosine geometry. Please justify this choice or compare with alternative fusion (e.g., addition or a learned projection).","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper has a promising direction and a useful new dataset, but the present version cannot be accepted because the reported numerical deltas are internally inconsistent (Table 2 vs abstract/introduction) and the RAG-based evaluation does not yet rule out label leakage through retrieved examples. The latter is a central concern for the paper's causal claim, not a cosmetic issue. I would encourage the authors to add retrieval-precision and label-flip controls and to correct the numbers; if those experiments support the claimed effect, a revised version would be within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: this paper contributes a genuinely new resource, AxiOM, 3,582 anxiety memes labeled into six GAD symptoms, plus a reproducible recipe (code and data released) that combines GPT-4o figurative reasoning with retrieval-augmented classification. The direction of the result, that injecting commonsense explanations and retrieved examples helps, is plausible and the paper gives credit where due: the dataset fills a real gap, the annotation process involved healthcare professionals, and the authors benchmark against 20 baseline variations including the prior SOTA on RESTORE. That alone makes it worth a look.\n\nThe soft spots are real but not all equally severe. The arithmetic is sloppy: intro and abstract quote deltas of 4.94% and 4.66% that do not match Table 2's own numbers (3.94 and 4.76 for those cells), and the ablation table in the appendix does not line up with the summary ablation in Table 2. Those are fixable reporting errors, but they lower confidence in the exact magnitudes. The post-hoc exclusion of the 'Lack of Energy' class from RESTORE is concerning and needs a sensitivity analysis; the human evaluation is only 40 samples and rates perceived quality, not correctness. Also, no error bars or variance across runs are reported, and the GPT-4o snapshot is unspecified.\n\nThe load-bearing concern, which the reader's stress-test correctly identifies, is the RAG module. Retrieved training examples are fed to the classifier with their gold labels, and retrieval operates in an embedding space built from OCR text plus GPT-4o reasoning that explicitly names the mental state. That creates a straightforward label-copying path: the nearest training memes for a test meme are likely to share the same symptom class, so the classifier can do well by copying the majority retrieved label. Near-duplicate memes across train/test from the same subreddits would make this even worse. The paper reports no retrieval precision, no random-or-label-flipped retrieval baseline, and no deduplication analysis. If that shortcut explains the 4-6 F1 point gains, the central claim that figurative-commonsense infusion is the driver is unsupported.\n\nThe paper is not circular in the mathematical sense, and the authors do honestly note some limitations of LLM reasoning. But the RAG shortcut is not acknowledged, and it is the first thing a referee should test.\n\nWho benefits: researchers working on mental-health meme classification, and anyone building small multimodal datasets for sensitive domains. They will get a solid new dataset and a clear baseline structure. The method itself is a combination of known components, so the value is mostly in the dataset and the empirical recipe.\n\nVerdict: worth a serious referee, but with the expectation of major revision. The numbers must be corrected, the RESTORE exclusion justified, and the RAG evaluation needs retrieval-precision checks, a label-flipped retrieval baseline, and a near-duplicate analysis. Until then, treat the reported gains as provisional. I would not cite the performance claims in their current form, but I would cite AxiOM if I needed an anxiety-meme benchmark.","headline":"Useful new anxiety-meme dataset and a plausible pipeline, but the RAG evaluation has a label-leakage risk that needs to be closed before the F1 gains can be trusted.","tokens_in":18617,"tokens_out":1673,"would_cite":true,"duration_ms":15711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting LLM-generated figurative-commonsense reasoning plus retrieved similar meme examples into a BART classifier is claimed to beat all baselines and the prior state of the art on fine-grained anxiety and depression meme classification.","keywords":["mental health memes","figurative language understanding","commonsense reasoning","anxiety symptom classification","depression symptom classification","retrieval-augmented generation","AxiOM dataset","multimodal classification"],"falsifier":"Take the AxiOM test set and replace GPT-4o's figurative reasoning with human-authored gold explanations, and separately with randomly shuffled explanations drawn from other memes, keeping retrieval fixed. If the shuffled-reasoning condition retains the full 4.20% weighted-F1 gain over OCR-only BART, then the content of the reasoning is not what drives the result; if the human-reasoning condition raises scores further, the LLM explanations are the bottleneck.","tokens_in":17522,"feed_emoji":"🧠","tokens_out":9549,"duration_ms":79167,"temperature":0.7,"pith_summary":"The paper sets out to show that fine-grained mental-health symptom labels can be read out of memes if the classifier is first given explicit figurative-commonsense knowledge, not just pixels and OCR text. To test this, the authors built AxiOM, a dataset of 3,582 memes labeled into six anxiety symptoms derived from the GAD questionnaire, and they propose M3H, which uses an LLM to generate structured reasoning about cause-effect, figurative understanding, and mental state for each meme, retrieves similar labeled examples from a retrieval-augmented database, and feeds all of this into a BART classifier. On AxiOM and on the public RESTORE depression dataset, M3H reports weighted-F1 gains of 4.20% and 4.66% over the prior state of the art, with larger macro-F1 gains. Ablation experiments show that removing the reasoning or retrieval modules hurts performance on both datasets. If the claim holds, it means explicit commonsense explanations are a practical way to improve machine understanding of figurative mental-health content.","feed_headline":"Commonsense reasoning lifts meme mental-health scores by 4–6%","feed_subtitle":"GPT-4o explanations plus retrieved examples beat 20 baselines on anxiety and depression meme classification.","key_machinery":"The pipeline has three stages. Stage one is figurative reasoning generation: an LLM is prompted to output three structured attributes for each meme—cause-effect relations, figurative understanding (metaphors, irony, symbols), and the depicted mental state. Stage two is knowledge fusion through retrieval-augmented generation: a sentence transformer creates embeddings of each training meme's OCR text and its reasoning, the two embeddings are concatenated, and for a new meme the top-n similar training instances are retrieved by cosine similarity. Stage three is classification: a BART encoder-decoder takes a prompt that includes the OCR text, the generated figurative reasoning, and the retrieved examples, and emits the symptom label, as a single softmax on AxiOM and per-label thresholded probabilities on RESTORE. The authors' argument is that the figurative reasoning supplies what the image-text pair by itself does not: the commonsense bridge between the meme's literal surface and its mental-health meaning.","core_discovery":"M3H's central claim is that combining LLM-generated figurative reasoning with retrieval-augmented exemplars and a strong text classifier gives the best reported results on mental-health meme symptom classification. On the new AxiOM anxiety dataset and the RESTORE depression dataset, the full M3H configuration reaches macro-F1 of 67.52 and 70.00 and weighted-F1 of 68.79 and 70.00, beating six baselines in 20 variations, including the prior state of the art, by 4.20% and 4.66% weighted-F1 respectively. The paper's ablation study shows that removing either the figurative-commonsense module or the retrieval module degrades performance on both datasets, which is the evidence that each component is load-bearing.","pith_inferences":["The authors do not test whether the content of the reasoning, rather than its mere presence, drives the gain; swapping the LLM's explanations for human-written ones or for deliberately vague ones would separate the content effect from the format effect.","Because the classifier consumes only OCR text, reasoning, and retrieved text, the image enters only indirectly through the LLM's reasoning; a testable extension is to ablate the image from the reasoning prompt to quantify how much visual information is actually being used.","The same recipe of LLM reasoning plus retrieval plus a strong text classifier is likely transferable to other figurative domains such as political cartoons or sarcastic product reviews, but the paper does not claim that and it would need its own dataset and baselines.","The paper explicitly disclaims diagnostic use; a downstream possibility, not a paper claim, would be to add a verification step that checks generated reasoning against the meme's visual content, since the authors list faithfulness of generated reasoning as future work."],"forward_implications":["If the claim holds, explicit LLM-generated commonsense reasoning is a portable ingredient that can be added to a text classifier to improve fine-grained mental-health classification without building a new multimodal architecture.","The AxiOM dataset fills a gap by providing a public benchmark for anxiety-meme classification with six GAD-based labels, where no dedicated public dataset existed before.","The retrieval module's contribution is separable and sizable: the reported ablation shows a jump of about 5 points when retrieval is added, so example-based knowledge infusion is doing real work beyond the reasoning text.","The error analysis identifies a concrete next target: confusion between 'Lack of Worry Control' and 'Impending Doom' shows that adjacent anxiety symptoms still need better disambiguation even when the model receives figurative reasoning."],"supporting_citations":[{"why":"Supplies the RESTORE depression-meme dataset and the prior state-of-the-art classifier that M3H must beat.","marker":"[45]"},{"why":"Supplies the LLM that generates the figurative-commonsense reasoning used in both retrieval and the classifier prompt.","marker":"[30]"},{"why":"Supplies the BART encoder-decoder architecture used as M3H's final classifier.","marker":"[21]"},{"why":"Supplies the best OCR-only baseline (Mental-BERT), the reference point against which the reasoning-plus-retrieval gain is measured.","marker":"[15]"},{"why":"Supplies the COMET commonsense transformer used as an alternative reasoning generator in the comparison.","marker":"[4]"},{"why":"Supplies the LLaVA multimodal model used as another alternative reasoning generator in the comparison.","marker":"[22]"}],"fun_headline_variants":["Commonsense reasoning boosts meme mental-health scores by 4-6%","Meme symptom classification improved 4.2% by commonsense infusion","M3H framework: commonsense + retrieval lifts meme anxiety detection","Mental health memes: GPT-4o commonsense reasoning aids classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on GPT-4o's automatically written explanations being accurate and faithful to what each meme actually conveys; if those explanations are confidently wrong, the reported gains could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Commonsense reasoning boosts meme mental-health scores by 4-6%","Meme symptom classification improved 4.2% by commonsense infusion","M3H framework: commonsense + retrieval lifts meme anxiety detection","Mental health memes: GPT-4o commonsense reasoning aids classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1436,"prompt_tokens":951,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":567,"tokens_out":485,"duration_ms":4829,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:23:54.600199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the AxiOM test set and replace GPT-4o's figurative reasoning with human-authored gold explanations, and separately with randomly shuffled explanations drawn from other memes, keeping retrieval fixed. If the shuffled-reasoning condition retains the full 4.20% weighted-F1 gain over OCR-only BART, then the content of the reasoning is not what drives the result; if the human-reasoning condition raises scores further, the LLM explanations are the bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RESTORE depression-meme dataset and the prior state-of-the-art classifier that M3H must beat."},{"cited_title":"Biomolecules 11, 8 (2021), 1093","cited_arxiv_id":null,"evidence_quote":"Supplies the BART encoder-decoder architecture used as M3H's final classifier."}],"review_version":1}