{"id":"b282e3d2-d50b-489e-8e0f-6e6aae090494","arxiv_id":"2411.14647","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Introduces ZNO-Vision, a 4,306-item Ukrainian multimodal exam benchmark, plus a translated VQA set and a 20-dish cuisine test, and finds only Gemini, Claude, and Qwen2-VL-72B beat the chance baseline.","lead":"A new benchmark, ZNO-Vision, tests AI models on over 4,300 Ukrainian university entrance exam questions that include images. The paper also measures how well vision-language models write Ukrainian captions, answer translated visual questions, and identify Ukrainian dishes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ZNO rankings may largely reflect prompt-format compliance, not Ukrainian ability: Pixtral's 0.000 and documented instruction-following failures leave the 'handful above baseline' claim unsecured.","rationale":"The paper's central claim is not merely that ZNO-Vision exists but that it yields a meaningful ranking in which 'only a handful of models performed above baseline.' The reader's weakest assumption points at the single shared prompt and regex parser, and the manuscript's own evidence supports that this is the most fragile link. Pixtral-12b scoring 0.000 across all categories is a red flag, since even much smaller models score non-zero; combined with Section 5's documented instruction-following failures, the parser confound is concrete and potentially dominant. The authors acknowledge prompt bias in Section 7, but an acknowledgment does not neutralize the threat to the headline result. The proposed lenient re-scoring is cheap, reproducible, and would decisively show whether the ranking survives a fairer extraction rule. Dataset non-release is a separate reproducibility concern, but the parser issue is more directly load-bearing for the empirical claim. Because the concern is testable and the paper already flags it, I do not move the verdict; the reader's CONDITIONAL remains appropriate, with the condition being that the authors demonstrate the ranking is robust to lenient or model-specific answer extraction.","tokens_in":21438,"tokens_out":2919,"duration_ms":33680,"concrete_test":"Re-run the ZNO test evaluation with a lenient answer extractor: scan the full decoded output for any token matching А/Б/В/Г or A/B/C/D, score by the first or most frequent letter, and also compute per-model format-compliance rates. Then compare the lenient ranking to Table 1. If Pixtral-12b moves from 0.000 to within the baseline range or above it, and/or if Llama-3.2-11B or Paligemma crosses the 0.219 first-choice baseline, the fixed-prompt/parser protocol is the dominant driver of the reported ordering. A complementary check is to rerun with a per-model few-shot prompt that demonstrates the exact bracket format, equalizing compliance before scoring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the ZNO evaluation protocol measures Ukrainian multimodal understanding rather than the ability to emit one exact bracketed-letter format. Section 5 reports that 'even high-performing models like GPT-4o and Gemini frequently failed to respond in expected format,' and Table 1 shows Pixtral-12b at 0.000 on every split, which is implausible as pure competence and is the signature of a parser failure. The scorer extracts the last bracketed letter via regex, so a model that answers correctly but writes 'Відповідь: B' or adds explanation after the bracket is scored wrong. Section 7 concedes that the shared prompt 'might introduce a bias in model comparison,' but this is precisely the central comparison. If format compliance varies systematically by model family, the headline 'only a handful performed above baseline' and the relative ordering in Table 1 partly measure instruction-following in Ukrainian, not ZNO-domain understanding. This is an external-validity risk rather than an internal inconsistency, but it is load-bearing for the paper's main empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ZNO-Vision, a multimodal Ukrainian-language benchmark of 4,306 multiple-choice questions derived from standardized Ukrainian university entrance exams, and reports evaluations of several proprietary and open-source vision-language models on it, finding that only a handful of models beat a first-choice baseline. It also evaluates image captioning on Multi30K-UK, compares VQA 2.0 performance in English and a GPT-4o-translated Ukrainian subset, and introduces UACUISINE, a small cultural benchmark on Ukrainian dishes, including a fine-tuning experiment for PaliGemma. The central claims are that the ZNO benchmark is a valid instrument for measuring Ukrainian multimodal understanding and that current models are far from strong performance outside English.","tokens_in":21637,"tokens_out":5386,"duration_ms":50533,"significance":"If the ZNO-Vision benchmark is valid and the evaluations are sound, this would be a genuinely useful resource: Ukrainian is a low-resource language for multimodal evaluation, and grounding the benchmark in an external standardized exam avoids the circularity that plagues model-generated benchmarks. The paper also makes concrete falsifiable claims (e.g., systematic performance degradation on Ukrainian VQA, improvements from fine-tuning on culturally specific data). However, the manuscript's value is currently limited by three issues: the dataset and code are not released, the ZNO evaluation protocol appears to confound format-following ability with domain understanding, and the UACUISINE fine-tuning experiment has a potential train/test overlap problem. These issues bear directly on the headline empirical claims, so the significance is conditional on their resolution.","major_comments":[{"comment":"The ZNO evaluation scores the last bracketed letter extracted by a regex from a single shared Ukrainian prompt, and Section 5 reports that even high-performing models frequently failed to follow the expected output format; Pixtral-12b scores 0.000 on every split and subject (Tables 1, 5, 6). This pattern suggests that the ranking may largely reflect instruction-following and format compliance rather than Ukrainian multimodal understanding, and the limitation note in Section 7 that the shared prompt 'might introduce a bias in model comparison' concedes exactly this risk for the paper's central comparison. To support the headline claim that 'only a handful of models performed above baseline,' the authors should provide per-model format-compliance rates, a manual or lenient rescoring of a representative sample, or results with multiple prompts and parsing strategies; without such evidence, the ZNO ranking is unsecured. The absence of multiple runs or confidence intervals around the ZNO accuracies, combined with temperature=1 sampling (Section 4), further weakens the comparison.","section":"§3 Evaluation Framework; §5 Results & Discussion; Table 1"},{"comment":"The central artifact, the ZNO-Vision benchmark, is not released: no URL, data sample, or evaluation code is provided anywhere in the paper. Because the paper's main contribution is a new benchmark and a comparative evaluation, the complete dataset with gold labels, the exact split definitions, the prompt templates, and the scoring scripts must be made publicly available (or provided as supplementary material) for the results to be reproducible and for the benchmark to be usable by others. As written, the benchmark exists only as a description.","section":"§3 Datasets & Methodology; §4 Experimental Setup"},{"comment":"The fine-tuning experiment is potentially circular: the UACUISINE benchmark is constructed from 20 dishes with 140 question-answer pairs (Section 3), yet the fine-tuning training set is described as 4,615 image-question-answer triplets from the UACUISINE training dataset, and no train/test split or overlap analysis is reported. If the evaluation set is drawn from the same dishes and augmented images used for training, the reported improvement from exact match 0.03 to 0.34 (Table 4) would demonstrate memorization rather than generalization. The authors must clarify the exact provenance of the training and evaluation examples and, ideally, evaluate the fine-tuned model on held-out dishes or a disjoint split.","section":"§3 UACUISINE; §5 Fine-tuning observations; Table 4"},{"comment":"The translated VQA-1000-UA set is produced entirely by GPT-4o, and the paper reports no human evaluation, back-translation checks, or inter-annotator agreement for translation quality. Since GPT-4o is itself one of the evaluated models, the reported Ukrainian-vs-English degradation in Table 3 may be inflated or distorted by translation artifacts; the Section 7 acknowledgment of this limitation is not accompanied by any diagnostic measure. The authors should either provide a translation-quality assessment (e.g., human adequacy judgments or a second translation source) or temper the claim that the degradation 'is a manifestation of cultural and linguistic bias by the models.'","section":"§3 VQA 2.0; §5 Performance degradation; Table 3"}],"minor_comments":[{"comment":"The manuscript contains several typos and informal expressions: 'Additionaly' in Section 1, 'de found' in the Table 1 caption, 'benchmank' in Section 7, 'extraxt' for 'extract,' and 'quadratic braces' for 'square brackets' in Section 3. These should be corrected for a journal submission.","section":"Throughout"},{"comment":"Table 1 omits models that appear in the appendix, notably Llama-3.2-90B-Vision-Instruct and Qwen2-VL-2B, without stating a selection criterion. The main results table should either include all evaluated models or explain why some are relegated to the appendix.","section":"Table 1 vs Appendix A"},{"comment":"Figure 1 shows subject categories such as Accounting and Finance, Economics, Law, and Political Science, but Table 7 lists a different set of categories with no mapping between them. The category taxonomy should be consistent across the figure, tables, and text.","section":"Figure 1 and Table 7"},{"comment":"The text says intersection match improved 'from 0% to 15%,' but Table 4 shows the highest IM is 0.30 (at 1 epoch) and the 4-epoch value is 0.15. Please correct the text to match the table, or clarify which epoch is being referenced.","section":"§5 Fine-tuning observations"},{"comment":"The references to 'eie, 2024' and 'Osv, 2024' are bare access URLs in the text with no formal citation details; they should be converted to standard bibliographic entries. The GQA reference cites 'Ainslie et al., 2023' but the cited paper appears to be about multi-query transformer models, not the compositional QA dataset, so the reference list should be checked for correctness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The benchmark addresses a real gap and the external grounding is a clear strength, but the evaluation protocol, the missing data release, and the UACUISINE overlap question all bear on the main claims and are fixable within the scope of a revision. I would not reject, but the authors need to provide substantial additional evidence or revise their claims. I would also suggest the editor ask for confirmation that the UACUISINE training-set description is accurate, as the numbers (20 dishes, 140 benchmark pairs, 4,615 training triplets) do not obviously add up without further explanation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ZNO-Vision is the real contribution here. 4,306 multiple-choice items from standardized Ukrainian university entrance exams, with images, across 12+ subjects, is exactly the kind of resource that's missing for mid-resource languages. The UACUISINE benchmark is small but thoughtful, and the critique of WorldCuisines' Ukrainian entries is fair and specific. I'd want this benchmark to exist regardless of the model rankings.\n\nWhat the paper does well: the benchmark is grounded in external expert-crafted exam data, which sidesteps the usual circularity. Splitting dev/val/test and giving a first-choice baseline are good practices. The fine-tuning experiment on PaliGemma is honest about being a pilot. The translated VQA comparison is standard but useful, and the cultural-misattribution observations are worth reading even if qualitative.\n\nWhere I'd push back: the evaluation protocol is the soft spot, and the stress-test note lands. One shared prompt plus a regex that grabs the last bracketed letter penalizes models that answer correctly but write differently. Pixtral scoring 0.000 on every split, plus the paper's own report that GPT-4o and Gemini often ignored the format, means the headline ordering in Table 1 partly measures format compliance. The authors concede this in Limitations, which counts in their favor, but for the central empirical claim the concession is not enough. Also: no code or data release, so the benchmark can't be inspected or reused; VQA degradation is confounded by GPT-4o translation; ZNO tables have no error bars; and the fine-tuning result should be labeled on-distribution, since train and eval come from the same 20 dishes. The Multi30K-UK section basically admits BLEU is uninformative there—fine as a pilot, but I'd drop the 'first evaluation study of multimodal text generation' phrasing.\n\nNet: this deserves peer review, but with a request for dataset/code release and a parser-fairness analysis before the rankings can be trusted. The benchmark artifact is worth more than the scores.","headline":"ZNO-Vision is a genuinely useful new Ukrainian multimodal benchmark; just don't treat the model ranking as a pure measure of Ukrainian ability until the evaluation protocol and data are released.","tokens_in":22185,"tokens_out":1826,"would_cite":true,"duration_ms":19205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Introducing ZNO-Vision, a 4,306-question Ukrainian multimodal benchmark, this paper finds that only three of the nine vision-language models tested beat the always-pick-first-choice baseline, and that targeted fine-tuning on Ukrainian…","keywords":["Ukrainian language","multimodal benchmark","vision-language models","ZNO exam","visual question answering","low-resource languages","cultural bias","fine-tuning"],"falsifier":"Re-run the ZNO evaluation with an output parser that tolerates verbose or free-form answers (for example, asking a human or an LLM judge to locate the intended choice), and check whether models that scored near zero, such as Pixtral-12b at 0.000, jump dramatically; if they do, the reported ranking largely reflects format compliance rather than Ukrainian understanding. A second check: translate the ZNO questions into English and verify that high scorers stay high and low scorers stay low; if the English version erases the gap, the benchmark is measuring language coverage, not multimodal reasoning.","tokens_in":21268,"feed_emoji":"📝","tokens_out":5176,"duration_ms":45325,"temperature":0.7,"pith_summary":"This paper introduces ZNO-Vision, a benchmark of over 4,300 questions drawn from Ukrainian standardized university entrance exams, with images and multiple-choice letter answers spanning math, physics, chemistry, history, languages, and more. It evaluates nine proprietary and open vision-language models and finds that only Gemini Pro, Claude 3.5 Sonnet, and Qwen2-VL-72B score above the 'always pick the first option' baseline of roughly 22 percent. Alongside this, the paper translates a portion of the VQA benchmark into Ukrainian, measures caption-generation quality on Multi30K-UK, and introduces UACUISINE, a small cultural benchmark about Ukrainian dishes. The broader point is that low- and mid-resource languages lack reliable multimodal evaluation, and the authors argue such benchmarks are needed to measure and improve models for Ukrainian speakers.","feed_headline":"Most AI vision models fail a Ukrainian exam benchmark","feed_subtitle":"Only three of nine models beat the guess-first baseline on 4,300 exam questions.","key_machinery":"The central object is ZNO-Vision itself: 4,306 exam questions, each an image plus a Ukrainian multiple-choice question whose answer is a single letter, split into dev, validation, and test subsets. The evaluation harness pairs this with a fixed Ukrainian prompt that asks the model to put its answer in square brackets, then extracts the last bracketed letter with a regular expression, mirroring the original exam's rule that only the answer strictly in the answer box counts. A second object, UACUISINE, adapts dish-identification questions from WorldCuisines, translates them into Ukrainian, and adds new ingredient, recipe, and taste or texture questions across 20 dishes, scored by exact match, word-set intersection, and BERTScore. The benchmark's design is what allows a direct, letter-level comparison across models.","core_discovery":"The paper's central claim is that ZNO-Vision is a valid instrument for estimating how well vision-language models handle image-grounded Ukrainian questions, and that current models perform poorly on it: on the test split Gemini Pro reaches 0.675 accuracy, Claude 3.5 Sonnet 0.643, and Qwen2-VL-72B 0.512, while the first-choice baseline is 0.219. The authors also find that fine-tuning a small model (Paligemma-3b) on Ukrainian cultural data raises its exact-match score on UACUISINE from 3 percent to 34 percent, and that this improves translated-Ukrainian VQA accuracy by about five points while leaving English VQA nearly unchanged. They interpret the wide gaps between English and Ukrainian results, together with observed code-switching and mislabeling of Ukrainian dishes as Russian, as evidence of linguistic and cultural bias in current vision-language models.","pith_inferences":["A consequence the authors leave implicit is that per-model prompting or answer extraction would probably compress the ranking; the paper's own instruction-following observations hint that format adherence is a large hidden component of the scores.","The same recipe, a standardized national exam plus a small cultural benchmark, could be transplanted to other mid-resource languages, giving each language a culturally grounded rather than translated evaluation.","A testable extension would be to compare ZNO-Vision scores against text-only versions of the same questions; if text-only models match the multimodal results, the benchmark is measuring language knowledge more than vision grounding.","The cultural misattribution results could be turned into a diagnostic for fairness audits: measuring how often a model reassigns a UNESCO-recognized dish to a neighboring culture may track broader geopolitical bias in training data."],"forward_implications":["ZNO-Vision can serve as a reusable testbed for tracking progress in Ukrainian multimodal understanding across future model releases.","Targeted fine-tuning on Ukrainian cultural content is shown to improve Ukrainian VQA without degrading English performance, pointing toward domain adaptation as a practical path for low-resource languages.","The large English-to-Ukrainian degradation on translated VQA indicates that translation-based benchmarking alone understates the gap; native benchmarks like ZNO-Vision complement translated ones.","The code-switching and Ukrainian-dish misattribution results suggest that training-data curation must address cultural correctness, not just language fluency.","The benchmark's STEM-heavy, OCR-dependent question mix means it also functions as a Ukrainian OCR diagnostic."],"supporting_citations":[{"why":"Supplies the ZNO exam as the source material for the benchmark questions.","marker":"eie, 2024"},{"why":"Provides the Osvita portal where the image-requiring exam questions were collected.","marker":"Osv, 2024"},{"why":"Multi30K-UK, the Ukrainian multimodal caption dataset used for caption-generation evaluation.","marker":"Saichyshyna et al., 2023"},{"why":"The VQA benchmark that the authors translate into Ukrainian to measure performance degradation.","marker":"Antol et al., 2015"},{"why":"Defines the VQA evaluation protocol and accuracy metric, which the authors keep for comparability.","marker":"Agrawal et al., 2015"},{"why":"WorldCuisines, the multilingual cuisine benchmark whose Ukrainian coverage the UACUISINE benchmark critiques and adapts.","marker":"Winata et al., 2024"},{"why":"The UNLP shared task showing ZNO questions are challenging for text-only LLMs, motivating the multimodal extension.","marker":"Romanyshyn et al., 2024"}],"fun_headline_variants":["Ukrainian exam benchmark stumps most vision AI","Vision models flunk Ukrainian language and culture test","New benchmark: AI models struggle with Ukrainian questions","Only 3 of 9 vision models top baseline on Ukrainian exam"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a single shared Ukrainian prompt followed by a last-bracket regex parser measures all models fairly, so differences in following the requested output format could shift the ranking as much as real Ukrainian-language ability.","fun_headline_variants_meta":{"raw":{"variants":["Ukrainian exam benchmark stumps most vision AI","Vision models flunk Ukrainian language and culture test","New benchmark: AI models struggle with Ukrainian questions","Only 3 of 9 vision models top baseline on Ukrainian exam"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2466,"prompt_tokens":911,"completion_tokens":1555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1493}},"tokens_in":527,"tokens_out":1555,"duration_ms":10707,"temperature":1.0,"reasoning_tokens":1493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:03:06.586524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ZNO evaluation with an output parser that tolerates verbose or free-form answers (for example, asking a human or an LLM judge to locate the intended choice), and check whether models that scored near zero, such as Pixtral-12b at 0.000, jump dramatically; if they do, the reported ranking largely reflects format compliance rather than Ukrainian understanding. A second check: translate the ZNO questions into English and verify that high scorers stay high and low scorers stay low; if the English version erases the gap, the benchmark is measuring language coverage, not multimodal reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The UNLP shared task showing ZNO questions are challenging for text-only LLMs, motivating the multimodal extension."}],"review_version":1}