{"id":"7775fdb6-46bf-4d47-b8bb-fdc63b92608b","arxiv_id":"2507.01702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AdamMeme is an adaptive, agent-based evaluation framework that iteratively refines meme text to expose model-specific weaknesses in multimodal models' understanding of meme harmfulness.","lead":"This paper introduces AdamMeme, a multi-agent framework that tests how well multimodal AI models understand harmful memes by repeatedly rewriting meme text into harder versions. It scores 11 vision-language models with AI judges and adapts the test set to each model's weaknesses, producing category-level strengths and failure profiles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'weakness' trigger in iterative refinement is conflated with ambiguity: refined memes are deliberately sanitized (Figure 12) and never human-validated as still harmful, so s'<s may measure disagreement about an underspecified meme, not a model-specific reasoning failure.","rationale":"The reader's CONDITIONAL verdict and weakest assumption focus on GPT-4o judge circularity (human exact-score agreement 56.7%). That is a real concern, and our worry overlaps with it, but the most load-bearing issue is arguably more fundamental: the construct validity of the 'weakness' signal itself. Even if scoring were perfectly uncircular, the refinement procedure intentionally removes explicit harmfulness cues and then interprets any score drop as a model-specific weakness without ever establishing that the refined meme is genuinely harmful. The paper's own human studies validate the quality of refined text (readability, coverage) but not its harmfulness ground truth; the low Fairness score (1.68, agreement 0.429) in Table 10 is an explicit red flag. This matters because the headline contribution is the fine-grained weakness analysis (e.g., Doubao-Lite's Race/Religion weaknesses, GPT-4o's Animal weakness), and all of those inferences flow through the s'<s trigger. The proposed concrete test—human harmfulness labeling of refined samples and comparison of human-FR with agent-FR—would settle whether the revealed 'weaknesses' are artifacts of ambiguity. If the test passes (humans agree refined samples are harmful and human-FR reproduces the pattern), the framework's central claim is substantially supported; if it fails, the claimed model-specific weaknesses are not established, and the paper's contribution reduces to an ill-specified benchmark generator. The verdict stays CONDITIONAL because the framework is well-described and the code is released, so the missing validation is obtainable rather than fundamentally impossible.","tokens_in":24237,"tokens_out":4696,"duration_ms":60312,"concrete_test":"Take all refined meme samples from the experiments, plus a matched set of original samples. Have independent human annotators (blinded to target-model responses, scores, and reference answers) label each meme as harmful or harmless under the same 8-category taxonomy. Then compute: (1) the human-harmful rate among refined samples where s'<s; if it is far below the human-harmful rate for original samples (or below a pre-registered threshold like 80%), the s'<s trigger is confounded with ambiguity; (2) a human-FR for each target model on the refined samples using the same threshold of 4.0, and check whether the model-specific weakness pattern in Tables 1-2 survives; if human-FR shows no such pattern (or high disagreement with GPT-4o FR), the central claim of model-specific weakness discovery fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AdamMeme reveals model-specific weaknesses in harmful-meme understanding. This requires that a drop in score after refinement (Algorithm 1, line 10: s' < s) indicates a genuine reasoning failure. That interpretation is unsupported because the refinement stage is explicitly designed to 'reduce the apparent harmfulness or controversial nature' of the meme text (Refiner prompt, Figure 12), and no human annotation step ever confirms that the refined memes remain harmful. The human evaluation in Appendix D assesses Redundancy, Diversity, Readability, Coverage, Fairness, and Suitability of refined texts (Table 10), but not whether the refined meme is actually harmful under the taxonomy. Notably, Fairness scores only 1.68/3 with inter-annotator agreement 0.429, indicating human ambiguity about the refined items. Without ground-truth harmfulness labels for refined samples, a lower score can simply reflect a reasonable alternative reading—e.g., in the Figure 5 case study, GPT-4o interprets the sanitized text as describing veterinarians treating a goat, and this benign reading receives a lower score only because the reference answer assumes a hidden harmful intent. The problem is not limited to judge bias: even a perfect, independent judge would penalize a model for failing to infer a harmfulness that the prompt design has rendered opaque. This directly undermines the framework's claim to expose 'specific limitations' and 'finer-grained weaknesses' (Section 1, Section 2.4), because the refined samples may be testing cue-removal robustness rather than harmfulness understanding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdamMeme, an agent-based evaluation framework designed to adaptively probe how well multimodal large language models (mLLMs) understand meme harmfulness. The framework first mines a harmfulness taxonomy and per-sample misbelief statements from unlabeled memes using multiple GPT-4o-based agents, then scores target models via a reference-based GPT-4o judge, and finally iteratively refines meme texts to create harder samples. The paper interprets a drop in a target model's score after refinement (s' < s) as evidence of a specific reasoning weakness. Experiments on 11 mLLMs report average scores and failure rates across eight harmfulness categories, with the authors claiming that the framework reveals varying model-specific strengths and weaknesses in fine-grained detail.","tokens_in":24595,"tokens_out":5451,"duration_ms":62880,"significance":"If the proposed framework's core inference is valid, the contribution is genuinely useful: the idea of dynamic, adaptive evaluation with misbelief-grounded refinement is a reasonable alternative to static accuracy-based benchmarks, and the authors ship code and include human studies, which is commendable. The paper also attempts reliability analysis of its scoring and mining stages, and the multi-agent design with taxonomy expansion is a plausible engineering contribution. However, the central claim that a refined sample's lower score exposes a model-specific reasoning failure is not currently supported, because the refinement step is explicitly designed to reduce apparent harmfulness and the resulting samples are never human-validated as still harmful. The evaluator dependence on GPT-4o and the low human agreement on scores further weaken the interpretation. The work is therefore a promising framework with significant validity gaps that need to be addressed before its conclusions can be accepted.","major_comments":[{"comment":"The central claim that s' < s exposes a model-specific weakness is load-bearing, but it is not supported by the current design. The Refiner prompt in Figure 12 explicitly instructs the agent to 'reduce the apparent harmfulness or controversial nature' of the meme text, yet no human annotation step confirms that the refined meme still conveys the same harmfulness category and misbelief. The human evaluation in Table 10 measures Redundancy, Diversity, Readability, Coverage, Fairness, and Suitability, but none of these criteria establishes that the refined sample remains harmful under the taxonomy; in fact, Coverage (2.06/3) and Fairness (1.68/3, agreement 0.429) are mediocre and suggest ambiguity about the refined content. The Figure 5 case study illustrates the problem: the target model reads the sanitized text as describing veterinarians treating a goat, and the lower score is attributed to a failure to infer hidden harmfulness that the prompt design has rendered opaque. Even a perfectly unbiased judge would penalize a model for not inferring harmfulness that the refined meme no longer clearly conveys. The authors should either add a human-annotated validity check for refined samples' continued harmfulness or temper the claim that score drops quantify reasoning failures.","section":"§2.4, Algorithm 1, Figure 12"},{"comment":"The evaluation loop is closed through a single model: GPT-4o generates reference answers, scores target responses, refines memes, and is itself one of the target models. The human agreement on exact scores is only 56.7% (Table 3), which is not sufficient to support the fine-grained score-drop comparisons underlying the main claims; the 73.8% agreement on failure classification is more robust, but the paper overstates this as 'high intra-class agreement.' The authors acknowledge evaluator bias in the Limitations section, but the acknowledgment does not address the concrete risk that target models using GPT-4o-distilled training data produce responses that are systematically penalized for style rather than for genuine reasoning errors. To make the claims credible, the paper should report agreement on the full score distribution, test the stability of the s'<s signal with an independent judge (e.g., a different LLM or a larger human sample), and show that the revealed 'weaknesses' are not artifacts of the judge's preferences.","section":"§2.3, §3.4, Table 3"},{"comment":"The iterative refinement loop is designed so that it continues only when s' < s and breaks on the first success. Consequently, the final history H is systematically enriched with samples on which the target model scores low, making the failure-rate increases in Table 2 partly mechanical. The improvement in FR without refinement is therefore not, by itself, evidence that the Refiner generates genuinely harder cases; it is a selection artifact. The authors should include a control condition that refines memes unconditionally (or that refines samples with score increases as well), and report the distribution of refined-sample scores separately from original-sample scores. Without such a control, the qualitative claim that refinement 'exposes weaknesses in a deeper understanding of harmfulness' is not distinguishable from the effect of sampling the difficulty distribution.","section":"Algorithm 1, lines 10–15"},{"comment":"The main results tables report no error bars or significance markers, despite the Appendix stating that 'compared results (p < 0.05 under t-test) are averaged over three random 3 runs.' The paper should report variances or confidence intervals and clarify which category-level and model-level comparisons are statistically significant. Additionally, the paper does not include a direct comparison with the static benchmark that motivates the framework (e.g., GOAT-Bench, cited as the main prior work on mLLM harmful-meme evaluation), so it is not demonstrated that AdamMeme's adaptive evaluation provides insights beyond accuracy-based benchmarks. A baseline comparison using the same target models on the static benchmark, with a discussion of what additional weaknesses are revealed only through refinement, would substantially strengthen the contribution.","section":"Tables 1–2, Appendix A"}],"minor_comments":[{"comment":"The heading 'Agent Human Agreement' should specify the metric (apparently exact-match percentage), and the term 'intra-class agreement' is likely intended to be 'inter-annotator agreement' or 'percentage agreement'; currently it is unclear what statistical quantity is reported.","section":"Table 3"},{"comment":"The Refiner prompt in Figure 12 says to 'preserve some aspects of the original context on the misbelief,' which is weaker than the text in §2.4 claiming that misb is used to 'ensure that the multimodal content should still retain the same false belief after modification.' This discrepancy should be reconciled, and the preservation requirement should be reflected in the prompt.","section":"Figure 12 vs §2.4"},{"comment":"Entries such as '00.50' and '08.54' appear to be formatting errors (stray leading zeros or missing decimal formatting) and should be corrected.","section":"Table 2"},{"comment":"The bullet claiming to be 'the first to evaluate mLLMs' ability to understand harmful memes from a model-centric, analytical perspective' is overstated given the existence of GOAT-Bench (Lin et al., 2024b), which also evaluates mLLMs on meme-based social abuse; the contribution should be phrased in terms of the adaptive and dynamic nature of the evaluation.","section":"§1, contributions"},{"comment":"The sentence about the t-test ('Compared results (p < 0.05 under t-test) are averaged over three random 3 runs') is grammatically unclear and does not specify whether the test is paired or unpaired, or how multiple comparisons across categories and models were handled.","section":"Appendix A"},{"comment":"The figure legend labels the y-axis as 'Avg. Score' but the caption could clarify that this is the average score across target models; error bars or confidence bands would be helpful given the lack of reported variance elsewhere in the paper.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's engineering contribution is real, but the central validity argument—that score drops after sanitizing refinement are evidence of model-specific reasoning failures—needs substantial additional support. The lack of human validation of refined memes' harmfulness, the closed loop through GPT-4o, and the selection artifact in the refinement loop are all fixable within the scope of a revision, so I do not recommend rejection. I would ask the editor to ensure the revision includes the proposed control experiments and independent-judge checks, and to watch that the claims are appropriately scaled back if those experiments are not feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely new: instead of a static accuracy benchmark, AdamMeme runs a closed loop that mines a harmfulness taxonomy, scores a target model with GPT-4o-based references, then rewrites meme text to make it harder and repeats. The code is public, the pipeline is clearly described, and the authors are upfront about the GPT-4o-evaluator issue in the Limitations. That counts for a lot.\n\nBut the central claim—that refinement exposes model-specific weaknesses in harmfulness understanding—does not hold up as cleanly as the paper presents it. The refiner prompt explicitly instructs the model to \"reduce the apparent harmfulness or controversial nature\" of the text (Figure 12). The human study of refined texts measures readability, diversity, coverage, and the like, but never checks whether the rewritten meme is still harmful under the taxonomy. The fairness score is low (1.68/3) with weak inter-annotator agreement. So the trigger s' < s in Algorithm 1 does not unambiguously mean the target model failed to grasp harmfulness. The goat case in Figure 5 is the clearest example: the sanitized text admits a perfectly benign reading, and the model's benign interpretation is scored lower only because the reference answer assumes a hidden abusive intent. A lower score there is penalizing the model for not inferring harmfulness that the prompt has rendered opaque. On top of that, the stopping rule keeps refining only while s' < s, so the refined set is composed of failures by construction; the FR increases in Table 2 are partly mechanical.\n\nThe judge circularity is real but acknowledged, and the human agreement numbers are moderate rather than damning. The more fixable issues are the missing error bars in the main tables and the absence of a direct comparison to a static benchmark on the same target models.\n\nThis is not a desk-reject. The protocol is useful as a dynamic probe, and the qualitative cases are thought-provoking. But the paper currently sells the quantitative output as evidence of model-specific reasoning failure, and that claim needs major revision. The authors should either get human annotators to verify that refined items remain harmful, or reframe the contribution as robustness to text sanitization rather than harmfulness understanding. Independent judges would also strengthen the case.\n\nSend it to peer review, but the reviewers should push on construct validity before the numbers are taken at face value.","headline":"Adaptive evaluation loop is a real step beyond static benchmarks, but the paper's load-bearing claim that refined samples expose model-specific harmfulness weaknesses is unsupported because the refiner deliberately sanitizes text and no human check confirms the rewrites remain harmful.","tokens_in":25126,"tokens_out":3600,"would_cite":true,"duration_ms":44484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewriting memes exposes hidden safety gaps in multimodal AI.","keywords":["multimodal large language models","harmful meme understanding","agent-based evaluation","adaptive benchmark","model-specific weakness","multi-agent system","safety evaluation","misbelief statement"],"falsifier":"Rescore the paper's 616 human-evaluated sample responses with human raters using the same rubric, then recompute per-model and per-category failure-rate rankings; if those rankings diverge materially from the GPT-4o-scorer rankings in Tables 1 and 2, the claimed model-specific weaknesses are partly artifacts of the judge. A more direct test: build a small scored set in which GPT-4o's own answers and answers from models trained on GPT-4o-distilled data are of equal human-judged quality, and check whether GPT-4o still assigns systematically higher scores to the GPT-4o-like answers.","tokens_in":24046,"feed_emoji":"🕵️","tokens_out":8785,"duration_ms":90426,"temperature":0.7,"pith_summary":"AdamMeme is an agent-based evaluation framework that claims to show where multimodal large language models fail when asked to explain why a meme is harmful. It starts from unlabeled memes, organizes them into harmfulness categories, scores each target model's open-form answer against reference answers, then iteratively rewrites the meme text into subtler variants that keep the same underlying harmful belief. The paper argues that this adaptive loop finds model-specific weaknesses that static, accuracy-based benchmarks miss because a model can handle familiar memes yet miss the same harm when explicit cues are removed. The reported results over eleven mLLMs show category-level differences, such as Doubao-Lite being most affected by refinement and GPT-4o being the most stable, and the authors present these as evidence that the framework provides fine-grained safety analysis rather than a single benchmark number.","feed_headline":"Rewriting memes exposes hidden safety gaps in multimodal AI","feed_subtitle":"AdamMeme scores 11 vision-language models, then edits meme text to expose failures static benchmarks miss.","key_machinery":"The load-bearing object is the misbelief statement, a concise natural-language sentence stating the generalized false belief that makes a meme harmful in a given category (for example, equating a public figure with a mythical creature as a dehumanizing stereotype). It is what lets the framework move from categories to finer-grained weaknesses: it is used as the retrieval key to find similar scored memes, as the constraint the Refiner must preserve while rewriting text, and as the label under which repeated failures are clustered. The mechanism that makes the evaluation adaptive is the scoring-refinement loop—score a meme, rewrite its text to hide the harm, rescore, and if the score drops, retrieve another sample with a similar misbelief and repeat—which the paper reports converges in about six iterations.","core_discovery":"The central claim is that harmful-meme understanding in mLLMs is not a single skill that one accuracy number can capture: it varies by harmfulness category and by how explicitly the meme states its stereotype or slur. AdamMeme therefore converts evaluation from a one-shot accuracy test into an adaptive probe. A set of mining agents builds a taxonomy of harmfulness categories, starting from Race, Gender, Religion, Nationality, Disability, and Animal and adding Political and Child Exploitation when warranted; a Narrator extracts a misbelief statement for each meme-category pair; scoring agents then grade target-model responses on a 1-10 scale against reference answers produced by a multi-candidate wisdom-of-crowds procedure; and a Refiner agent rewrites meme text to make the same misbelief harder to spot. When the refined meme receives a lower score, that specific misbelief-context combination is recorded as a weakness. The paper's experimental results, reported per category and per model in Tables 1 and 2, show that different models have different failure-rate patterns—Step-series models struggle most with Disability, Doubao-Lite degrades most under refinement, GPT-4o changes least—which is the evidence for model-specific weaknesses.","pith_inferences":["If the GPT-4o-judge bias is real, the reported gap between GPT-4o and other models may be partly self-preference; a fair test would compare against human scores on the same responses rather than only reporting agreement rates.","The refined samples that lower a model's score are, by construction, adversarial examples a model has not learned from; they could be recycled as training data to harden safety reasoning.","The same misbelief-anchored loop could be read as a risk-profile generator: category-level failure rates plus misbelief topics predict which real-world memes a deployed model is likely to mishandle.","Because the Refiner removes explicit cues while preserving the misbelief, a model that still flags the harm is doing genuine multimodal reasoning; this suggests refinement difficulty could be used as a measure of reasoning depth independent of raw accuracy."],"forward_implications":["Static accuracy benchmarks will understate safety gaps, because models can pass familiar memes while failing on subtle variants of the same harmful belief.","Model rankings depend on harmfulness category, so a single average score hides which categories a model handles poorly.","Because the framework scores open-form reasoning rather than binary labels, it can distinguish shallow cue-matching from deeper multimodal understanding.","The refinement loop converges in roughly six iterations, suggesting a bounded number of adaptive rounds is enough to reach a stable weakness profile.","The framework can audit new memes without human annotation, which makes safety evaluation cheaper and easier to keep current as memes evolve."],"supporting_citations":[{"why":"Supplies the FHM raw meme data and the hateful-meme task framing used as input to the framework.","marker":"Kiela et al. (2020)"},{"why":"Supplies the HarM raw meme dataset and the notion of harmful meme targets that informs the initial taxonomy.","marker":"Pramanick et al. (2021)"},{"why":"Supplies the MAMI raw meme data and misogyny-related harmfulness categories used for sampling.","marker":"Fersini et al. (2022)"},{"why":"The static accuracy-based meme safety benchmark that AdamMeme explicitly contrasts with in motivation and experiments.","marker":"Lin et al. (2024b)"},{"why":"Provides the reference-based LLM-as-a-judge practice that the Scorer agent is built on.","marker":"Zheng et al. (2023)"},{"why":"Supplies the six seed categories (race, gender, religion, nationality, disability, animal) for the initial harmfulness taxonomy.","marker":"Cao et al. (2023)"},{"why":"Provides the definition of harmful memes that the paper adopts as its conceptual foundation.","marker":"Sharma et al. (2022)"}],"fun_headline_variants":["Adaptive meme edits expose per-model safety gaps","AdamMeme stress-tests multimodal AI with rewritten memes","Probing memes reveals each model's blind spot for harm","Model-specific meme weaknesses exposed by adaptive test","Rewriting memes uncovers individual mLLM safety limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that GPT-4o, the model that writes the reference answers and scores every target model, is a neutral and reliable judge of all eleven target models, including itself; the paper reports only 56.7 percent exact-score agreement between GPT-4o and human evaluators.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive meme edits expose per-model safety gaps","AdamMeme stress-tests multimodal AI with rewritten memes","Probing memes reveals each model's blind spot for harm","Model-specific meme weaknesses exposed by adaptive test","Rewriting memes uncovers individual mLLM safety limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1405,"prompt_tokens":958,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":574,"tokens_out":447,"duration_ms":5124,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:45:32.698916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rescore the paper's 616 human-evaluated sample responses with human raters using the same rubric, then recompute per-model and per-category failure-rate rankings; if those rankings diverge materially from the GPT-4o-scorer rankings in Tables 1 and 2, the claimed model-specific weaknesses are partly artifacts of the judge. A more direct test: build a small scored set in which GPT-4o's own answers and answers from models trained on GPT-4o-distilled data are of equal human-judged quality, and check whether GPT-4o still assigns systematically higher scores to the GPT-4o-like answers.","supporting_citations":[],"review_version":1}