Pith. sign in

REVIEW 4 major objections 7 minor 6 references

MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MEENA is a Persian multimodal exam dataset of about 7,500 original questions and 3,000 English translations, with metadata on difficulty, human success, and traps, built to test vision-language models on native Persian scientific and…

desk verdict Original Persian multimodal exam benchmark with rich metadata; the evaluation numbers, though, are only as solid as an unvalidated LLM answer-extraction pipeline. read the letter →

arxiv 2508.17290 v1 pith:Y5CKUFYO submitted 2025-08-24 cs.AI cs.LG

Omid Ghahroodi , Arshia Hemmat , Marzia Nouri , Seyed Mohammad Hadi Hosseini , Doratossadat Dastgheib , Mohammad Vali Sanian , Alireza Sahebi , Reihaneh Zohrabi
show 3 more authors
Mohammad Hossein Rohban Ehsaneddin Asgari Mahdieh Soleymani Baghshah
This is my paper · ORCID
classification cs.AIcs.LG
keywords Persianlanguagevision-languagemodelsmultimodalbenchmarkeducationalexamscross-lingualevaluationhallucinationdetectionmultiple-choicequestionslow-resourceNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents MEENA (PersianMMMU), a dataset of about 7,500 original Persian multiple-choice exam questions with images plus a 3,000-question English translation, drawn from Iran's school system from primary through upper secondary level. The authors' aim is to give vision-language models a native Persian benchmark that covers scientific reasoning, mathematics, physics, charts, diagrams, and Persian art, rather than relying on translated English material. They argue this fills a gap left by earlier Persian VLM benchmarks, which lacked scientific reasoning tasks or reused translated content. They also show how the dataset's metadata—difficulty, educational level, human success rate, trap flags—can turn benchmark scores into diagnostics of where models fail, and they report experiments that vary prompting and image availability to separate knowledge, reasoning, and visual grounding.

What carries the argument

The central object is the MEENA dataset itself, organized by where images appear (question only, choices only, or both) and annotated with difficulty, educational level, trap presence, and human success rates. The evaluation mechanism is a controlled image-manipulation protocol: each model is run in zero-shot, in-context learning, first-describe, wrong-image, and without-image settings, so that the contribution of the visual input can be isolated. Model outputs are turned into answers by a two-stage extractor that first applies regex patterns and then asks GPT-4o-mini, acting as a judge, to infer the selected option or flag a missing, wrong, or unintelligible image. The translation pipeline uses GPT-4o with an LLM-as-a-judge quality filter, which is what allows the same questions to be compared across Persian and English.

What would settle it

Take a random sample of the 3,067 English questions, have native Persian speakers translate them back to Persian independently, and compare the back-translations with the originals; if a substantial share change meaning, difficulty, or image dependence, the cross-lingual accuracy gaps reported in the paper would not be clean evidence about model ability. A separate check would compare the GPT-4o-mini answer extractor against human scoring on a few hundred model outputs; disagreement would cast doubt on the reported accuracies.

Watch

Extended reading notes

Core claim

MEENA is the first Persian multimodal dataset aimed at scientific and art exam evaluation, containing roughly 7,483 multiple-choice questions, of which 6,936 come from the Pellekan Yadgiri platform and 547 from curated online sources including national university entrance exams. Each question is tied to an image, and the dataset records educational level, difficulty, answer explanations, trap indicators, student success rate, subject, and creation year. A parallel English subset of 3,067 questions was created by translating the Persian items with GPT-4o and filtering them with an LLM-as-a-judge semantic score. The paper's central claim is that this combination of original Persian content and rich metadata supports reliable evaluation of Persian VLM abilities, and its experiments find that knowledge-based questions are answered more accurately than reasoning questions, that accuracy drops when the correct image is replaced or removed, and that models vary widely in detecting image mismatches.

Load-bearing premise

MEENA's English half is produced by machine translation and kept only if a GPT-4o judge scores it 4 or higher, so the paper assumes that process preserves meaning and difficulty well enough for any Persian-English performance gap to count as a model capability difference rather than a translation artifact.

Editorial extensions

If this is right

  • Persian VLM developers gain a native benchmark whose questions were not translated from English, so good performance reflects genuine Persian understanding rather than overlap with English training data.
  • Because every question has an English twin, the benchmark provides a paired cross-lingual test: a gap between Persian and English accuracy on the same item is a direct measure of language transfer cost.
  • The trap and difficulty metadata lets a user separate perception failures from reasoning failures: a model that collapses on trap questions has a different weakness from one that fails uniformly.
  • The wrong-image and without-image settings turn the benchmark into a hallucination probe, quantifying how often a model notices an irrelevant or missing image.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same recipe were applied to other under-resourced languages, each would need original school exams and native cultural content; machine translation alone would not reproduce MEENA's cultural grounding.
  • The reported Persian-English gaps are probably conservative, since the English side is machine-translated and filtered by an LLM judge; high-quality human translation might narrow the gap by removing translation artifacts.
  • A direct test of the first-describe finding would compare models on questions where the image is genuinely needed versus decorative; the paper's aggregate numbers leave that distinction open.
  • Because MEENA's questions are drawn from public educational sources, the dataset could double as a contamination monitor: a model that unexpectedly excels on the Persian subset may have seen those exact items during training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MEENA (PersianMMMU), a bilingual Persian-English multiple-choice benchmark for vision-language models, assembled from Iranian educational platforms and national exam sources, with about 7,500 Persian and 3,000 English questions, image-based questions, and metadata including difficulty, human success rate, and trap indicators. The authors evaluate five models (GPT-4o, GPT-4o-mini, GPT-4-Turbo, Gemini-2.0-flash, and InstructBLIP-T5) under five settings: zero-shot, in-context learning, first-describe, wrong-image, and without-image. They report that knowledge-based tasks outperform reasoning tasks, that Gemini-2.0-flash is more robust at detecting image mismatches, that GPT-4-Turbo and GPT-4o rarely miss image presence, and that performance declines on harder chemistry and mathematics questions.

Significance. If the benchmark is sound, it fills a genuine gap: there is no existing Persian multimodal benchmark with original, curriculum-grounded scientific and reasoning exam questions and rich metadata. The release of data, code, and a leaderboard is valuable for the community, and the bilingual design plus the wrong-image and without-image settings are useful for diagnosing whether models actually use visual information. The paper also ships transparent prompts and appendixed per-category results, which supports reproducibility. However, the central experimental conclusions are currently only as reliable as the unvalidated answer-extraction and hallucination-classification pipeline, so the empirical claims should be treated as provisional pending validation.

major comments (4)
  1. [§4.3] The two-stage answer-extraction pipeline (regex followed by GPT-4o-mini as a judge) is never validated. The paper reports no human-annotated sample, no agreement rate, no confusion matrix, and no distribution of the 'unknown' category, even though the prompt in Appendix D shows that the judge outputs four decision types that directly feed accuracy, 'no image', 'wrong image', and 'cannot understand' counts. Because GPT-4o-mini is itself one of the evaluated models, outputs of the evaluated system are scored by a same-family judge. All numbers in Table 2 and Figures 2 and 3, including the headline claims about knowledge versus reasoning and about Gemini's mismatch detection, are downstream of this pipeline. The authors should validate the extractor on a stratified sample of model outputs, report per-category accuracy, and show that headline conclusions are robust when only regex-extracted answers are used.
  2. [§4.2 and Figure 3] The 'hallucination detection rate' (Figure 3) is defined by the judge's classification of a response as 'wrong image' or 'no image', not by an independent measure of whether the model identified the mismatch. This makes the hallucination conclusion circular with respect to the unvalidated judge. In addition, the construction of the wrong-image pool is described only as images 'confirmed to be unrelated'; the confirmation protocol, the number of distinct wrong images, and the matching procedure between questions and wrong images are not specified, so the difficulty of the wrong-image condition is not reproducible. Please provide the pool construction details and a human-checked sample of judge decisions in the wrong-image setting.
  3. [§3.4] The English subset is produced by GPT-4o translation followed by GPT-4o-as-a-judge scoring with a retention threshold of 4/5, and no human evaluation of translation quality is reported. The English data are also selected from the Persian data by translating 'items with Persian text in images' and retaining only those with pure English or non-text visuals, which could introduce selection bias in favor of image-light or text-light questions. Consequently, the cross-lingual performance gaps reported in Section 5 and Figures 2 and 9 may reflect translation or selection artifacts rather than model capability. The authors should report a human-validated translation-quality sample, the distribution of judge scores, and the number of questions removed by each filtering stage.
  4. [Table 2 and Section 5] Experimental results are reported from a single run with no confidence intervals, standard errors, or significance tests. Several comparisons that support the main claims are small (for example, English zero-shot accuracy of 0.474 for GPT-4o versus 0.494 for Gemini-2.0-flash, and many within-model differences across settings are a few points), so the reported rankings and the 'knowledge > reasoning' gap of 10-19% need error bars or repeated runs to establish that the differences are not noise. Please add uncertainty estimates or a statistical comparison over bootstrap samples of questions.
minor comments (7)
  1. [§3.1] The sampling formula appears as '1/weight1/4'; this should be typeset as 1/weight^(1/4) so the exponent is clear.
  2. [Table 1] The header contains a typo, 'Descripive Answer' should be 'Descriptive Answer', and the column 'Trap # Tasks' is missing a space.
  3. [Figure 3] The y-axis is labeled 'Detection Rate' but the values are raw counts reaching 1,400; please relabel as count or convert to a proportion with the appropriate denominator.
  4. [Section 5] The sentence 'the detection rate difference ... is over 400 detections' mixes counts with rates; please report rates or proportions consistently.
  5. [Table 2] The asterisk footnote is too terse: it should state explicitly that InstructBLIP-T5 does not support the ICL and WO settings, and the reader should not have to infer this from the table.
  6. [Title and Section 6] The term 'N-level' in the title is never defined; either define the educational level taxonomy (LP, UP, LS, US) in the title context or rephrase the title.
  7. [Appendix C] Some values in Tables 3 and 4 and Figures 9 and 13 are not fully aligned with Table 2, such as the missing Persian Wrong-Image entry for GPT-4-Turbo; please reconcile the main table and the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark's reported accuracies are empirical measurements, and no predicted quantity is fitted from its own inputs.

full rationale

MEENA is a dataset-construction and evaluation paper, not a derivation paper. The Persian questions come from external exam sources, the English subset is produced by GPT-4o translation with an LLM-as-a-judge filter, and the experiments measure model accuracy under five prompting settings. There is no equation in which a claimed result is equivalent to an input by construction, no fitted parameter is later renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work. The overlapping-author citation to Ghahroodi et al. (2024) is background related work, not the justification for any experimental conclusion. The answer-extraction pipeline using regex and GPT-4o-mini as a judge is a measurement-validity concern, and the hallucination detection rate is operationally defined by the judge's classification, but this is the standard mechanics of benchmark evaluation rather than a circular reduction: the accuracies and rankings are not forced by the definition of the extraction step, because the judge's labels could in principle diverge from the models' true selections and the paper simply does not validate that divergence. Similarly, the GPT-4o translation filter affects which English items are retained, but the paper does not derive cross-lingual performance from that filter. No specific circular step can be quoted from the manuscript, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of exam accuracy as a measure, on the reliability of the translation and answer-extraction pipeline, and on the comparability of educational levels across languages. The hand-chosen sampling exponent and translation threshold are curation choices that affect dataset balance.

free parameters (2)
  • Category sampling exponent = 1/4
    Used to weight sampling across categories to address imbalance; chosen by the authors, not fitted to an external target; affects the category distribution of the final dataset.
  • Translation retention threshold = 4 (on 1-5 scale)
    Samples with LLM-judge quality scores below 4 were revised or rejected; this threshold sets the trade-off between translation quality and dataset size.
assumptions (3)
  • domain assumption Multiple-choice accuracy on exam questions is a valid proxy for VLM understanding.
    The entire benchmark treats accuracy on 4-option questions as the measure of capability; this assumes the questions are unambiguous and the difficulty levels are meaningful.
  • ad hoc to paper The GPT-4o-based translation and GPT-4o-mini-based answer extraction are sufficiently accurate for evaluating other models.
    The English subset and the evaluation pipeline depend on these model-based tools; if they are biased, the reported cross-lingual and hallucination results could be distorted.
  • domain assumption The educational levels of Iran's 12-year system (LP, UP, LS, US) provide a meaningful ordering of difficulty for both Persian and translated English questions.
    The analysis stratifies by level, so the assumption is that levels correspond to comparable cognitive demands across languages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment." pith.science (2026). https://pith.science/paper/Y5CKUFYO

@misc{pith2026250817290,
  author       = {Pith},
  title        = {Pith review of: MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5CKUFYO}},
  note         = {Machine review of arXiv:2508.17290}
}
read the original abstract

Recent advancements in large vision-language models (VLMs) have primarily focused on English, with limited attention given to other languages. To address this gap, we introduce MEENA (also known as PersianMMMU), the first dataset designed to evaluate Persian VLMs across scientific, reasoning, and human-level understanding tasks. Our dataset comprises approximately 7,500 Persian and 3,000 English questions, covering a wide range of topics such as reasoning, mathematics, physics, diagrams, charts, and Persian art and literature. Key features of MEENA include: (1) diverse subject coverage spanning various educational levels, from primary to upper secondary school, (2) rich metadata, including difficulty levels and descriptive answers, (3) original Persian data that preserves cultural nuances, (4) a bilingual structure to assess cross-linguistic performance, and (5) a series of diverse experiments assessing various capabilities, including overall performance, the model's ability to attend to images, and its tendency to generate hallucinations. We hope this benchmark contributes to enhancing VLM capabilities beyond English.

Figures

Figures reproduced from arXiv: 2508.17290 by the authors.

Figure 1
Figure 1. Overview of dataset and some sample questions from different tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Accuracy comparison of reasoning and knowledge-based tasks across models in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Hallucination detection rates across three vision-language models (GPT-4 Mini, [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Heatmap of model accuracy across different levels of the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of model accuracy across different levels of the [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Sample of MEENA questions 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Sample of MEENA questions with picture in choices [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Sample of MEENA questions including Persian texts in picture [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Farsi and English performance across different experiments and [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Comparison of Farsi and English performance across different experiments and [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of each model across experiments on the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of each model across experiments on the [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of each model across experiments on the [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Performance comparison of each model across experiments on the [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 canonical work pages

  1. [1]

    Arab Tribe - Khuzestan

    work page

  2. [2]

    Baharlu Tribe - Fars

    work page

  3. [3]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P

    URL https://arxiv.org/abs/2411.12591. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P . Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685. Lianghui Zhu, Xinggang Wang, and Xinlo...

    arXiv 2023

  4. [4]

    Turk - Hamedan Figure 6: Sample of MEENA questions 15 Preprint. Question زیر،درکدامگزینهبهصورتصحیحآمدهاست؟مجموعانگشتانبازدستراستوانگشتانبازدستچپشکل Which option correctly states the sum of the openfingers of the right hand and the open fingers of theleft hand in the figure below? Question: پشتبرگههاکاملاًسفیداست.ازکدامشکلگستردهمکعبیبانمایروبهروحاصلمیشود؟ ...

    work page arXiv 2024

  5. [2021]

    Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang

    URL https://arxiv.org/abs/2103.00020. Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings, 2024. URL https://arxiv.org/abs/ 2305.02317. Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu ...

    arXiv 2024

  6. [2024]

    doi: 10.3389/frai.2024.1430984

    ISSN 2624-8212. doi: 10.3389/frai.2024.1430984. URL https://www.frontiersin. org/journals/artificial-intelligence/articles/10.3389/frai.2024.1430984. Arshia Hemmat, Adam Davies, Tom Lamb, Jianhao Yuan, Philip Torr, Ashkan Khakzar, and Francesco Pinto. Hidden in plain sight: Evaluating abstract shape recognition in vision-language models. Advances in Neura...

    arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.