Pith. sign in

REVIEW 3 major objections 3 minor 32 references

Although its full text is a different benchmark paper, the abstract claims fine-tuned GPT-4o-mini and Gemini Flash 2.5 win the MAHED 2025 Arabic hope/hate/emotion tasks with macro F1 up to 79.6%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The submission cannot be reviewed as a coherent paper: its abstract and full text are two different papers, so the abstract's claims have no supporting body.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The submission package is broken: the abstract describes an Arabic shared-task paper, but the full text is a different paper on scientific-journal covers, so the reported MAHED results have no derivable support in this record. the 3 major comments →

arxiv 2508.15810 v1 pith:W3B5JGGS submitted 2025-08-15 cs.CL cs.AIcs.LG

Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

Nouar AlDahoul , Yasir Zaki This is my paper
classification cs.CL cs.AIcs.LG
keywords Arabic NLPhate speech detectionhope detectionemotion detectionmultimodal memesLLM fine-tuningMAHED 2025content moderation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that fine-tuned large language models beat both base LLMs and pre-trained embedding models on three Arabic-language tasks—detecting hope, hate/offensive speech, and emotion—in both plain text and memes, using the MAHED 2025 challenge data. Its headline claim is that GPT-4o-mini fine-tuned on Arabic text and Gemini Flash 2.5 fine-tuned on Arabic memes achieve macro F1 scores of up to 72.1%, 57.8%, and 79.6% on the three tasks, securing first place overall. A reader should care because Arabic content moderation is under-resourced, and a winning recipe built from off-the-shelf commercial models would be immediately usable. However, the full text supplied under this title is a different paper, describing a benchmark called MAC for multimodal scientific understanding; none of the experiments behind the abstract's numbers appear in it. The abstract's claim therefore cannot be checked from the body as submitted.

Core claim

The discovery the authors report is that task-specific fine-tuning of commercial LLMs outperforms both base LLMs and pre-trained embedding classifiers on Arabic hope, hate/offensive, and emotion detection, and that the best model differs by modality: GPT-4o-mini fine-tuned on Arabic textual speech leads for text tasks, while Gemini Flash 2.5 fine-tuned on Arabic memes leads for multimodal meme tasks. They report macro F1 scores of 72.1%, 57.8%, and 79.6% for tasks 1, 2, and 3, and first place in the MAHED 2025 challenge. The manuscript body provided here, however, is the text of a different paper (MAC-2025, a live benchmark for multimodal scientific understanding), so the experiments, datase

What carries the argument

The central mechanism is modality-matched fine-tuning: taking a general-purpose pretrained LLM and adapting it on task-specific Arabic text or Arabic memes rather than using it zero-shot. The abstract credits this mechanism for the reported macro-F1 gains and the first-place finish, claiming that the fine-tuned commercial models extract the signal that base LLMs and pre-trained embedding models miss. The full text as supplied does not describe this mechanism or its experiments.

Load-bearing premise

The claim collapses if the reported macro-F1 numbers were not produced by the described fine-tuned GPT-4o-mini and Gemini Flash 2.5 models on the official MAHED 2025 held-out test set, because the submitted full text contains no experiments to support them.

What would settle it

Run the three MAHED 2025 tasks on the official test split with the same fine-tuned model versions and the same evaluation script; the claim fails if the macro F1 scores do not reproduce the reported 72.1%, 57.8%, and 79.6% within normal variance. A simpler check available now: the submission's own full text is a different paper, so the manuscript itself contains no task-specific result tables to verify.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuned commercial LLM APIs could serve as ready Arabic content moderation systems, avoiding the need to train models from scratch.
  • Text and meme detection would be best treated as separate pipelines, since the winning models differ by modality.
  • The reported 79.6% macro F1 top score would mark the current ceiling for automatic Arabic hope/hate/emotion detection with commercial API models on the MAHED data.
  • On shared challenges like MAHED, the competitive edge would shift from architecture design to fine-tuning configuration, data selection, and evaluation strategy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the reported scores hold, the practical bottleneck likely shifts to annotation quality and class balance, since macro F1 is sensitive to label distribution; a testable extension is per-class error analysis to identify which hate and emotion categories remain confused.
  • Editorial inference: Because the winning models are commercial APIs, the exact scores may not be reproducible later as model versions change; a useful extension is to record API versions and checkpoints at evaluation time.
  • Editorial inference: The same fine-tuning recipe would plausibly transfer to dialectal or code-switched Arabic social media, and could be tested on Egyptian or Maghrebi datasets to measure generalization beyond the MAHED split.
  • Editorial inference: The mismatch between the abstract and the supplied full text suggests the method description may live in a separate report; if the MAC text is a misattached file, the challenge-system paper could still be released with training details and per-task breakdowns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The abstract of this submission claims that fine-tuned GPT-4o-mini (for Arabic textual speech) and Gemini Flash 2.5 (for Arabic memes) achieve macro-F1 scores of 72.1%, 57.8%, and 79.6% on tasks 1, 2, and 3 of the ArabicNLP MAHED 2025 challenge, and that the proposed system secured first place overall. The supplied full text, however, is a different paper: 'MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding' (arXiv:2508.15802), which proposes a benchmark of scientific journal covers and evaluates MLLMs on image-to-text and text-to-image matching. This full text contains no Arabic datasets, no mention of MAHED, no hate-speech/hope/emotion tasks, no fine-tuning configurations, and no results tables reporting macro-F1 scores. The central empirical claim of the abstract is therefore unsupported by any experimental description in the submitted artifact.

Significance. If the claimed results were substantiated, they would be of practical value for Arabic content moderation and would provide evidence that fine-tuned commercial LLMs can perform strongly on an external, community-organized benchmark. A positive feature is that the evaluation target (MAHED 2025) is external to the authors, so the claimed outcome is not defined circularly in terms of the method itself. However, the significance cannot currently be assessed because the submitted manuscript body is a coherent but unrelated benchmark paper. No supporting experimental details, dataset splits, hyperparameters, evaluation protocol, or code are present in the record. Thus the claimed contribution is, at this stage, an abstract-level assertion without a verifiable basis.

major comments (3)
  1. [Abstract vs. full-text manuscript] The abstract reports the paper's only occurrence of the three macro-F1 scores (72.1%, 57.8%, 79.6%) and the 'first place overall' claim. The supplied full text is arXiv:2508.15802, 'MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding.' This body does not mention Arabic, MAHED, GPT-4o-mini, Gemini Flash 2.5, hope, hate speech, offensive language, or emotions. No table or equation in the full text reports macro-F1 or any result for tasks 1-3. The central claim of the paper is therefore entirely unverified by the submitted artifact.
  2. [Full text, §4 and Tables 1-8] The experimental sections evaluate models on MAC-2025 scientific-cover matching using accuracy, ECE, NLL, and RMS. There are no training splits, fine-tuning hyperparameters, prompt templates, evaluation scripts, or held-out test details for the Arabic tasks described in the abstract. Even if the full text were read as the intended paper, it would not support the premise that the reported MAHED scores come from the official evaluation protocol on a held-out test set without information leakage. This is a load-bearing gap: the abstract's numbers cannot be checked or reproduced.
  3. [Manuscript identity and internal consistency] The submission's title page identifies the full text as a COLM 2025 paper by Jiang et al. with arXiv:2508.15802, while the abstract describes a different MAHED 2025 system paper. This is not a minor presentation issue; it means the document as submitted is internally inconsistent. The authors (or the editorial office) need to supply the canonical manuscript matching the abstract before the claimed results can receive substantive review.
minor comments (3)
  1. [Abstract] The challenge name is spelled inconsistently: 'MAHED 2025' appears in one sentence and 'Mahed 2025' in another. Please standardize.
  2. [Full text, figures and captions] Several figure captions and text passages in the supplied full text contain placeholder glyphs (e.g., '����'), indicating an encoding or rendering problem. If this manuscript is to be considered, these should be repaired.
  3. [General] Once the correct manuscript is supplied, the authors should add a reproducibility appendix listing the training/validation splits, fine-tuning budgets, model versions, and the exact evaluation protocol used for the MAHED 2025 test set.

Circularity Check

0 steps flagged

No circularity; the abstract's MAHED 2025 results are unsupported by the mismatched full text, but the evaluation target is external, so no definitional or fitted-input circularity is present.

full rationale

The submitted record is a mismatch: the abstract describes fine-tuning GPT-4o-mini and Gemini Flash 2.5 on the MAHED 2025 Arabic text/meme tasks and reports macro-F1 scores and a first-place finish, while the supplied full text is an unrelated COLM 2025 paper (arXiv:2508.15802) on the MAC benchmark for multimodal scientific understanding. None of the abstract's experiments, training splits, hyperparameters, evaluation protocol, or result tables appear in the full text. This makes the central empirical claim unverifiable from the supplied record, but unverifiability is not circularity. The MAHED benchmark is external to the authors and the reported scores are not defined in terms of the proposed method; no fitted parameter is relabeled as a prediction, and no equation equates the output with an input. Considering the full text on its own terms, the MAC paper evaluates MLLMs on externally sourced journal-cover data and proposes DAD as an inference-time wrapper; its claims rest on benchmark accuracy, calibration metrics, and ablations, not on self-citations or definitional identities. No specific circular reduction can be quoted from the text, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The abstract names no fitted constants, so the free_parameters list is empty; fine-tuning hyperparameters mentioned nowhere are not enumerated. The central claims rest on the validity of the external MAHED 2025 benchmark and its protocol, and on the contradicted premise that the supplied body documents the claimed experiments. No invented entities are introduced: the abstract proposes no new components, only fine-tuning of existing models.

axioms (2)
  • domain assumption The MAHED 2025 dataset labels and official evaluation protocol are valid and were used to compute the reported macro-F1 scores.
    The abstract claims specific F1 numbers and a first-place finish without specifying the train/test split or scoring script; the supplied body does not document them, so the protocol premise is unverifiable.
  • ad hoc to paper The supplied full text corresponds to the paper described in the abstract.
    This premise is contradicted by the record itself: the full text is the MAC benchmark paper, arXiv 2508.15802, by different authors. The abstract's claims therefore have no documented experimental basis in the submitted document.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models." pith.science (2026). https://pith.science/paper/W3B5JGGS

@misc{pith2026250815810,
  author       = {Pith},
  title        = {Pith review of: Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3B5JGGS}},
  note         = {Machine review of arXiv:2508.15810}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rise of social media and online communication platforms has led to the spread of Arabic textual posts and memes as a key form of digital expression. While these contents can be humorous and informative, they are also increasingly being used to spread offensive language and hate speech. Consequently, there is a growing demand for precise analysis of content in Arabic text and memes. This paper explores the potential of large language models to effectively identify hope, hate speech, offensive language, and emotional expressions within such content. We evaluate the performance of base LLMs, fine-tuned LLMs, and pre-trained embedding models. The evaluation is conducted using a dataset of Arabic textual speech and memes proposed in the ArabicNLP MAHED 2025 challenge. The results underscore the capacity of LLMs such as GPT-4o-mini, fine-tuned with Arabic textual speech, and Gemini Flash 2.5, fine-tuned with Arabic memes, to deliver the superior performance. They achieve up to 72.1%, 57.8%, and 79.6% macro F1 scores for tasks 1, 2, and 3, respectively, and secure first place overall in the Mahed 2025 challenge. The proposed solutions offer a more nuanced understanding of both text and memes for accurate and efficient Arabic content moderation systems.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    hidden language

    Results reveal a consistent performance gap between the Image2Text and Text2Image tasks. Specifically, the average accuracy across models for Image2Text is 21.6%, with an average rank of 0.237, whereas Text2Image achieves a substantially higher average accuracy of 38.7% and a lower average rank of 0.194. This indicates that retrieving matching images from...

    work page 2025

  3. [6]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793,

    Pith/arXiv arXiv

  4. [8]

    Deep anomaly detection with outlier exposure

    Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606,

    Pith/arXiv arXiv

  5. [9]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749,

    Pith/arXiv arXiv

  6. [10]

    Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model

    Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model. arXiv preprint arXiv:2406.11193,

    Pith/arXiv arXiv

  7. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  8. [13]

    Compbench: A comparative reasoning benchmark for multimodal llms

    Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Compbench: A comparative reasoning benchmark for multimodal llms. arXiv preprint arXiv:2407.16837,

    Pith/arXiv arXiv

  9. [14]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision- language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision- language models. arXiv preprint arXiv:2403.00231, 2024a. Shengzhi Li and Nima Tajbakhsh. Scigraphqa: A large-scale synthetic multi-turn question- answering dataset for scientific graphs. ar...

    Pith/arXiv arXiv

  10. [15]

    Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension. InAI for Accelerated Materials Design-Vienna 2024, 2024b. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tu...

    work page 2024

  11. [16]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL ������������������������������������������������������ . 12 Published as a conference paper at COLM 2025 Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, ...

    Pith/arXiv arXiv 2025

  12. [18]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    Pith/arXiv arXiv

  13. [19]

    Mmiu: Multimodal multi-image understanding for evaluating large vision-language models

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718,

    Pith/arXiv arXiv

  14. [20]

    Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan

    URL ������������������� ���������������������. Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for multimodal question answering on scientific papers. arXiv preprint arXiv:2407.09413,

    Pith/arXiv arXiv

  15. [21]

    Making monolingual sentence embeddings multilingual using knowledge distillation

    Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11

    work page 2020

  16. [22]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar

    URL ������������������������. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  17. [23]

    Milebench: Benchmarking mllms in long context

    Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532,

    Pith/arXiv arXiv

  18. [24]

    Introducing step1o, March 2024a

    StepFun. Introducing step1o, March 2024a. URL ���������������������������������. StepFun. Introducing step1v, March 2024b. URL ���������������������������������. 13 Published as a conference paper at COLM 2025 Jiamin Su, Yibo Yan, Fangteng Fu, Han Zhang, Jingheng Ye, Xiang Liu, Jiahao Huo, Huiyu Zhou, and Xuming Hu. Essayjudge: A multi-granular benchmark ...

    Pith/arXiv arXiv 2025

  19. [25]

    Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530,

    Pith/arXiv arXiv

  20. [26]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al

    URL ��������������������������������������. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arX...

    Pith/arXiv arXiv

  21. [27]

    Muirbench: A comprehensive benchmark for robust multi-image understanding

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411, 2024a. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-c...

    Pith/arXiv arXiv

  22. [28]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314,

    Pith/arXiv arXiv

  23. [29]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414,

    Pith/arXiv arXiv

  24. [30]

    We strictly ensure that all content included in the MAC com- plies with the open access or public use policies stated by each publisher

    14 Published as a conference paper at COLM 2025 A Open Access License of the Journals All cover images and their corresponding cover stories used in the construction of Multi- modal Academic Cover benchmark (MAC) are sourced from the official websites of leading scientific journals which published by Nature Portfolio, Science/AAAS, Cell Press and American...

    work page 2025

  25. [32]

    pills” and a “prescription pad

    misiden- tified the correct answer. While these state-of-the-art MLLMs successfully recognized the key visual elements in option C—namely, “pills” and a “prescription pad”—demonstrating strong perceptual capabilities, they subsequently reasoned that the image lacked content related to “drug resistance” or “mechanisms of cancer therapy.” Instead, both mode...

    work page 2025

  26. [2017]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    11 Published as a conference paper at COLM 2025 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv 2025

  27. [2020]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et ...

    Pith/arXiv arXiv

  28. [2021]

    Figureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, ´Akos K´ad´ar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300,

    Pith/arXiv arXiv

  29. [2022]

    Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255,

    Pith/arXiv arXiv

  30. [2023]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425,

    Pith/arXiv arXiv

  31. [2024]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al

    URL ������������������������������� ����������������� . Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923,

    Pith/arXiv arXiv

  32. [2025]

    Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg

    URL ������������������������. Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10424,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.