{"id":"dc09804f-e9b8-415a-a5e4-eb4e1f78a821","arxiv_id":"2508.17290","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MEENA is a new bilingual (Persian-English) image-based exam benchmark for evaluating vision-language models across scientific, reasoning, and cultural topics.","lead":"The paper introduces MEENA (PersianMMMU), a benchmark of about 7,500 Persian multiple-choice exam questions with images, plus a translated English subset, designed to test vision-language models on science, reasoning, and Persian culture. It reports that current models such as GPT-4o and Gemini-2.0-flash score below 0.5 accuracy on Persian questions and struggle more with reasoning than with knowledge recall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer extraction via regex plus GPT-4o-mini is unvalidated, and it is the same model family as an evaluated system; all headline accuracies and hallucination conclusions depend on this pipeline.","rationale":"The reader's strongest claim concerns dataset novelty, but the paper's central argument also includes extensive experiments and hallucination findings as stated contributions. I focus on answer extraction because it affects every reported accuracy, including Persian-only results, whereas translation quality affects only cross-lingual comparisons. The absence of any human validation for the extraction judge is a concrete correctness risk: the judge is a same-family model as one evaluated system, and the paper's headline conclusions are computed from its labels. This reinforces the reader's conditional verdict rather than overturning it, because the dataset itself may still be valuable and the extraction issue is fixable with a validation study.","tokens_in":21298,"tokens_out":6250,"duration_ms":69345,"concrete_test":"Manually annotate a stratified random sample of about 400 raw model responses (strata: model, language, experiment, and question-image case) with two independent annotators using the same label set as the prompt in Appendix D (1/2/3/4, no image, cannot understand, wrong image, unknown). Compare with the automated extraction, report Cohen's kappa and per-stratum agreement, then recompute Table 2 accuracies and Figure 3 detection rates using the human labels. If any per-model accuracy shifts by more than 2-3 points absolute, or if any model ranking flips, the reported results should be revised; if agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 describes a two-stage answer extraction: regex-based pattern matching, which fails in roughly half of cases, followed by GPT-4o-mini 'as a judge' that infers the selected option or classifies the response as no image / cannot understand / wrong image / unknown. The paper reports no validation of this judge: no human-annotated sample, no agreement rate, no confusion matrix, and no distribution of the 'unknown' category. Because GPT-4o-mini is itself one of the evaluated models, its outputs are scored by a same-family judge, and every number in Table 2 and Figures 2-3 (including the headline 'knowledge > reasoning' and 'Gemini detects image mismatches better' findings) is downstream of this pipeline. A systematic labeling bias—e.g., preferring early options, misreading Persian phrasing, or failing to recognize different formulations of 'no image'/'wrong image'—would change accuracies and model rankings without any change in model behavior. The 'hallucination detection rate' is defined by this classifier, not by an independent measure, so the paper's main experimental conclusions are only as reliable as the unstated accuracy of this extraction step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MEENA (PersianMMMU), a bilingual Persian-English multiple-choice benchmark for vision-language models, assembled from Iranian educational platforms and national exam sources, with about 7,500 Persian and 3,000 English questions, image-based questions, and metadata including difficulty, human success rate, and trap indicators. The authors evaluate five models (GPT-4o, GPT-4o-mini, GPT-4-Turbo, Gemini-2.0-flash, and InstructBLIP-T5) under five settings: zero-shot, in-context learning, first-describe, wrong-image, and without-image. They report that knowledge-based tasks outperform reasoning tasks, that Gemini-2.0-flash is more robust at detecting image mismatches, that GPT-4-Turbo and GPT-4o rarely miss image presence, and that performance declines on harder chemistry and mathematics questions.","tokens_in":21538,"tokens_out":3662,"duration_ms":39808,"significance":"If the benchmark is sound, it fills a genuine gap: there is no existing Persian multimodal benchmark with original, curriculum-grounded scientific and reasoning exam questions and rich metadata. The release of data, code, and a leaderboard is valuable for the community, and the bilingual design plus the wrong-image and without-image settings are useful for diagnosing whether models actually use visual information. The paper also ships transparent prompts and appendixed per-category results, which supports reproducibility. However, the central experimental conclusions are currently only as reliable as the unvalidated answer-extraction and hallucination-classification pipeline, so the empirical claims should be treated as provisional pending validation.","major_comments":[{"comment":"The two-stage answer-extraction pipeline (regex followed by GPT-4o-mini as a judge) is never validated. The paper reports no human-annotated sample, no agreement rate, no confusion matrix, and no distribution of the 'unknown' category, even though the prompt in Appendix D shows that the judge outputs four decision types that directly feed accuracy, 'no image', 'wrong image', and 'cannot understand' counts. Because GPT-4o-mini is itself one of the evaluated models, outputs of the evaluated system are scored by a same-family judge. All numbers in Table 2 and Figures 2 and 3, including the headline claims about knowledge versus reasoning and about Gemini's mismatch detection, are downstream of this pipeline. The authors should validate the extractor on a stratified sample of model outputs, report per-category accuracy, and show that headline conclusions are robust when only regex-extracted answers are used.","section":"§4.3"},{"comment":"The 'hallucination detection rate' (Figure 3) is defined by the judge's classification of a response as 'wrong image' or 'no image', not by an independent measure of whether the model identified the mismatch. This makes the hallucination conclusion circular with respect to the unvalidated judge. In addition, the construction of the wrong-image pool is described only as images 'confirmed to be unrelated'; the confirmation protocol, the number of distinct wrong images, and the matching procedure between questions and wrong images are not specified, so the difficulty of the wrong-image condition is not reproducible. Please provide the pool construction details and a human-checked sample of judge decisions in the wrong-image setting.","section":"§4.2 and Figure 3"},{"comment":"The English subset is produced by GPT-4o translation followed by GPT-4o-as-a-judge scoring with a retention threshold of 4/5, and no human evaluation of translation quality is reported. The English data are also selected from the Persian data by translating 'items with Persian text in images' and retaining only those with pure English or non-text visuals, which could introduce selection bias in favor of image-light or text-light questions. Consequently, the cross-lingual performance gaps reported in Section 5 and Figures 2 and 9 may reflect translation or selection artifacts rather than model capability. The authors should report a human-validated translation-quality sample, the distribution of judge scores, and the number of questions removed by each filtering stage.","section":"§3.4"},{"comment":"Experimental results are reported from a single run with no confidence intervals, standard errors, or significance tests. Several comparisons that support the main claims are small (for example, English zero-shot accuracy of 0.474 for GPT-4o versus 0.494 for Gemini-2.0-flash, and many within-model differences across settings are a few points), so the reported rankings and the 'knowledge > reasoning' gap of 10-19% need error bars or repeated runs to establish that the differences are not noise. Please add uncertainty estimates or a statistical comparison over bootstrap samples of questions.","section":"Table 2 and Section 5"}],"minor_comments":[{"comment":"The sampling formula appears as '1/weight1/4'; this should be typeset as 1/weight^(1/4) so the exponent is clear.","section":"§3.1"},{"comment":"The header contains a typo, 'Descripive Answer' should be 'Descriptive Answer', and the column 'Trap # Tasks' is missing a space.","section":"Table 1"},{"comment":"The y-axis is labeled 'Detection Rate' but the values are raw counts reaching 1,400; please relabel as count or convert to a proportion with the appropriate denominator.","section":"Figure 3"},{"comment":"The sentence 'the detection rate difference ... is over 400 detections' mixes counts with rates; please report rates or proportions consistently.","section":"Section 5"},{"comment":"The asterisk footnote is too terse: it should state explicitly that InstructBLIP-T5 does not support the ICL and WO settings, and the reader should not have to infer this from the table.","section":"Table 2"},{"comment":"The term 'N-level' in the title is never defined; either define the educational level taxonomy (LP, UP, LS, US) in the title context or rephrase the title.","section":"Title and Section 6"},{"comment":"Some values in Tables 3 and 4 and Figures 9 and 13 are not fully aligned with Table 2, such as the missing Persian Wrong-Image entry for GPT-4-Turbo; please reconcile the main table and the appendix.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a useful contribution and the authors have been transparent about prompts and data sources. The main risk is not novelty but measurement validity: the headline empirical claims all pass through an unvalidated GPT-4o-mini judge, and the cross-lingual analysis passes through an unvalidated GPT-4o translation judge. I would like to see human validation of both pipelines and uncertainty estimates before final acceptance; these are substantial but fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look. The paper's real contribution is the dataset: roughly 7,500 original Persian exam questions with images, spanning math, physics, chemistry, reasoning, and art, with metadata including school level, difficulty, trap flags, human success rates, and explanations. That genuinely fills a gap; prior Persian VLM benchmarks were translated, generated, captioning/OCR-only, or text-only. The bilingual subset of about 3,000 English questions is a useful extra, though it is machine-translated.\n\nThe paper does several things well. Data compilation is described with enough detail to be reproducible: HTML extraction, image merging, category sampling weighted by 1/weight^(1/4), and deduplication. The metadata is richer than anything else in Persian VLM evaluation, enabling human-vs-model comparisons by difficulty and trap questions. The experimental design (zero-shot, ICL, first-describe, wrong-image, without-image) is sensible, and the wrong-image/without-image settings are a good way to probe image dependence.\n\nSoft spots, in order of importance. First, the answer-extraction pipeline in Section 4.3 is unvalidated, and it is load-bearing. Regex catches about half the cases; the rest go to GPT-4o-mini as a judge, which labels the selected option or returns \"no image\"/\"cannot understand\"/\"wrong image\"/\"unknown.\" No human-annotated sample, no agreement rate, no confusion matrix, no breakdown of the \"unknown\" category. Every number in Table 2 and the hallucination-detection figures flows through this step. Since GPT-4o-mini is also an evaluated model, a systematic labeling bias could shift accuracies and model rankings without any change in model behavior. The \"hallucination detection\" conclusion is essentially this classifier's opinion. This is fixable: annotate a few hundred outputs and report agreement. Until then, the experimental conclusions are conditional.\n\nSecond, the English subset is translated by GPT-4o, filtered by GPT-4o-as-judge with a 4/5 retention threshold. Same-model self-check, no human evaluation of translation quality or difficulty preservation. Fine for a v1, but cross-lingual comparisons should be read as preliminary.\n\nThird, no confidence intervals or significance tests anywhere. Differences of a few points between models are reported as findings.\n\nMinor: the dataset and code URLs are promised but not given in the text; the appendix has empty table cells and a couple of implausible zeros for gpt-4o-mini that need checking. The novelty claim is plausible given the cited prior work.\n\nBottom line: the dataset deserves to exist and will likely be used. The evaluation section needs validator work before I would trust the model-ranking and hallucination claims. Send it to a serious referee; there is enough original, reproducible material here, but expect heavy revision on the evaluation methodology.","headline":"Original Persian multimodal exam benchmark with rich metadata; the evaluation numbers, though, are only as solid as an unvalidated LLM answer-extraction pipeline.","tokens_in":22086,"tokens_out":2702,"would_cite":true,"duration_ms":28162,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEENA is a Persian multimodal exam dataset of about 7,500 original questions and 3,000 English translations, with metadata on difficulty, human success, and traps, built to test vision-language models on native Persian scientific and…","keywords":["Persian language","vision-language models","multimodal benchmark","educational exams","cross-lingual evaluation","hallucination detection","multiple-choice questions","low-resource NLP"],"falsifier":"Take a random sample of the 3,067 English questions, have native Persian speakers translate them back to Persian independently, and compare the back-translations with the originals; if a substantial share change meaning, difficulty, or image dependence, the cross-lingual accuracy gaps reported in the paper would not be clean evidence about model ability. A separate check would compare the GPT-4o-mini answer extractor against human scoring on a few hundred model outputs; disagreement would cast doubt on the reported accuracies.","tokens_in":21130,"feed_emoji":"🎓","tokens_out":7909,"duration_ms":73533,"temperature":0.7,"pith_summary":"This paper presents MEENA (PersianMMMU), a dataset of about 7,500 original Persian multiple-choice exam questions with images plus a 3,000-question English translation, drawn from Iran's school system from primary through upper secondary level. The authors' aim is to give vision-language models a native Persian benchmark that covers scientific reasoning, mathematics, physics, charts, diagrams, and Persian art, rather than relying on translated English material. They argue this fills a gap left by earlier Persian VLM benchmarks, which lacked scientific reasoning tasks or reused translated content. They also show how the dataset's metadata—difficulty, educational level, human success rate, trap flags—can turn benchmark scores into diagnostics of where models fail, and they report experiments that vary prompting and image availability to separate knowledge, reasoning, and visual grounding.","feed_headline":"7,500 native Persian exam questions now test vision-language AI","feed_subtitle":"MEENA pairs Persian and English multiple-choice items with images to measure reasoning, traps, and hallucination.","key_machinery":"The central object is the MEENA dataset itself, organized by where images appear (question only, choices only, or both) and annotated with difficulty, educational level, trap presence, and human success rates. The evaluation mechanism is a controlled image-manipulation protocol: each model is run in zero-shot, in-context learning, first-describe, wrong-image, and without-image settings, so that the contribution of the visual input can be isolated. Model outputs are turned into answers by a two-stage extractor that first applies regex patterns and then asks GPT-4o-mini, acting as a judge, to infer the selected option or flag a missing, wrong, or unintelligible image. The translation pipeline uses GPT-4o with an LLM-as-a-judge quality filter, which is what allows the same questions to be compared across Persian and English.","core_discovery":"MEENA is the first Persian multimodal dataset aimed at scientific and art exam evaluation, containing roughly 7,483 multiple-choice questions, of which 6,936 come from the Pellekan Yadgiri platform and 547 from curated online sources including national university entrance exams. Each question is tied to an image, and the dataset records educational level, difficulty, answer explanations, trap indicators, student success rate, subject, and creation year. A parallel English subset of 3,067 questions was created by translating the Persian items with GPT-4o and filtering them with an LLM-as-a-judge semantic score. The paper's central claim is that this combination of original Persian content and rich metadata supports reliable evaluation of Persian VLM abilities, and its experiments find that knowledge-based questions are answered more accurately than reasoning questions, that accuracy drops when the correct image is replaced or removed, and that models vary widely in detecting image mismatches.","pith_inferences":["If the same recipe were applied to other under-resourced languages, each would need original school exams and native cultural content; machine translation alone would not reproduce MEENA's cultural grounding.","The reported Persian-English gaps are probably conservative, since the English side is machine-translated and filtered by an LLM judge; high-quality human translation might narrow the gap by removing translation artifacts.","A direct test of the first-describe finding would compare models on questions where the image is genuinely needed versus decorative; the paper's aggregate numbers leave that distinction open.","Because MEENA's questions are drawn from public educational sources, the dataset could double as a contamination monitor: a model that unexpectedly excels on the Persian subset may have seen those exact items during training."],"forward_implications":["Persian VLM developers gain a native benchmark whose questions were not translated from English, so good performance reflects genuine Persian understanding rather than overlap with English training data.","Because every question has an English twin, the benchmark provides a paired cross-lingual test: a gap between Persian and English accuracy on the same item is a direct measure of language transfer cost.","The trap and difficulty metadata lets a user separate perception failures from reasoning failures: a model that collapses on trap questions has a different weakness from one that fails uniformly.","The wrong-image and without-image settings turn the benchmark into a hallucination probe, quantifying how often a model notices an irrelevant or missing image."],"supporting_citations":[{"why":"Defines the MMMU English college-exam benchmark whose task structure MEENA adapts to Persian school exams.","marker":"Yue et al. (2024)"},{"why":"The earlier Persian VLM benchmark whose limitations—translated content, missing scientific reasoning, closed source—MEENA is designed to overcome.","marker":"Farsi et al. (2025)"},{"why":"Introduces Khayyam (PersianMMLU), the text-only Persian benchmark that MEENA extends with visual and multimodal questions.","marker":"Ghahroodi et al. (2024)"},{"why":"Describes GPT-4o, the model used for Persian-to-English translation and as one of the main evaluated VLMs.","marker":"Hurst et al. (2024)"},{"why":"Supplies the LLM-as-a-judge methodology used for translation filtering and for extracting answer options from model outputs.","marker":"Zheng et al. (2023)"},{"why":"Introduces InstructBLIP, the open vision-language model included as a comparison point in the evaluations.","marker":"Dai et al. (2023b)"}],"fun_headline_variants":["MEENA: first Persian benchmark for VLM reasoning and art","PersianMMMU: 7,500 Persian exam questions challenge VLMs","MEENA: bilingual exam dataset tests VLMs on Persian and English","New Persian benchmark MEENA reveals VLM image attention gaps","Persian VLM benchmark: 7,500 questions plus 3,000 English"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"MEENA's English half is produced by machine translation and kept only if a GPT-4o judge scores it 4 or higher, so the paper assumes that process preserves meaning and difficulty well enough for any Persian-English performance gap to count as a model capability difference rather than a translation artifact.","fun_headline_variants_meta":{"raw":{"variants":["MEENA: first Persian benchmark for VLM reasoning and art","PersianMMMU: 7,500 Persian exam questions challenge VLMs","MEENA: bilingual exam dataset tests VLMs on Persian and English","New Persian benchmark MEENA reveals VLM image attention gaps","Persian VLM benchmark: 7,500 questions plus 3,000 English"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1356,"prompt_tokens":918,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":534,"tokens_out":438,"duration_ms":4848,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:06:01.116449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 3,067 English questions, have native Persian speakers translate them back to Persian independently, and compare the back-translations with the originals; if a substantial share change meaning, difficulty, or image dependence, the cross-lingual accuracy gaps reported in the paper would not be clean evidence about model ability. A separate check would compare the GPT-4o-mini answer extractor against human scoring on a few hundred model outputs; disagreement would cast doubt on the reported accuracies.","supporting_citations":[],"review_version":1}