Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Introducing ZNO-Vision, a 4,306-question Ukrainian multimodal benchmark, this paper finds that only three of the nine vision-language models tested beat the always-pick-first-choice baseline, and that targeted fine-tuning on Ukrainian…

desk verdict ZNO-Vision is a genuinely useful new Ukrainian multimodal benchmark; just don't treat the model ranking as a pure measure of Ukrainian ability until the evaluation protocol and data are released. read the letter →

arxiv 2411.14647 v1 pith:TZBYZ73T submitted 2024-11-22 cs.CL

Yurii Paniv , Artur Kiulian , Dmytro Chaplynskyi , Mykola Khandoga , Anton Polishko , Tetiana Bas , Guillermo Gabrielli This is my paper · ORCID
classification cs.CL
keywords Ukrainianlanguagemultimodalbenchmarkvision-languagemodelsZNOexamvisualquestionansweringlow-resourcelanguagesculturalbiasfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ZNO-Vision, a benchmark of over 4,300 questions drawn from Ukrainian standardized university entrance exams, with images and multiple-choice letter answers spanning math, physics, chemistry, history, languages, and more. It evaluates nine proprietary and open vision-language models and finds that only Gemini Pro, Claude 3.5 Sonnet, and Qwen2-VL-72B score above the 'always pick the first option' baseline of roughly 22 percent. Alongside this, the paper translates a portion of the VQA benchmark into Ukrainian, measures caption-generation quality on Multi30K-UK, and introduces UACUISINE, a small cultural benchmark about Ukrainian dishes. The broader point is that low- and mid-resource languages lack reliable multimodal evaluation, and the authors argue such benchmarks are needed to measure and improve models for Ukrainian speakers.

What carries the argument

The central object is ZNO-Vision itself: 4,306 exam questions, each an image plus a Ukrainian multiple-choice question whose answer is a single letter, split into dev, validation, and test subsets. The evaluation harness pairs this with a fixed Ukrainian prompt that asks the model to put its answer in square brackets, then extracts the last bracketed letter with a regular expression, mirroring the original exam's rule that only the answer strictly in the answer box counts. A second object, UACUISINE, adapts dish-identification questions from WorldCuisines, translates them into Ukrainian, and adds new ingredient, recipe, and taste or texture questions across 20 dishes, scored by exact match, word-set intersection, and BERTScore. The benchmark's design is what allows a direct, letter-level comparison across models.

What would settle it

Re-run the ZNO evaluation with an output parser that tolerates verbose or free-form answers (for example, asking a human or an LLM judge to locate the intended choice), and check whether models that scored near zero, such as Pixtral-12b at 0.000, jump dramatically; if they do, the reported ranking largely reflects format compliance rather than Ukrainian understanding. A second check: translate the ZNO questions into English and verify that high scorers stay high and low scorers stay low; if the English version erases the gap, the benchmark is measuring language coverage, not multimodal reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that ZNO-Vision is a valid instrument for estimating how well vision-language models handle image-grounded Ukrainian questions, and that current models perform poorly on it: on the test split Gemini Pro reaches 0.675 accuracy, Claude 3.5 Sonnet 0.643, and Qwen2-VL-72B 0.512, while the first-choice baseline is 0.219. The authors also find that fine-tuning a small model (Paligemma-3b) on Ukrainian cultural data raises its exact-match score on UACUISINE from 3 percent to 34 percent, and that this improves translated-Ukrainian VQA accuracy by about five points while leaving English VQA nearly unchanged. They interpret the wide gaps between English and Ukrainian results, together with observed code-switching and mislabeling of Ukrainian dishes as Russian, as evidence of linguistic and cultural bias in current vision-language models.

Load-bearing premise

The evaluation assumes that a single shared Ukrainian prompt followed by a last-bracket regex parser measures all models fairly, so differences in following the requested output format could shift the ranking as much as real Ukrainian-language ability.

Editorial extensions

If this is right

  • ZNO-Vision can serve as a reusable testbed for tracking progress in Ukrainian multimodal understanding across future model releases.
  • Targeted fine-tuning on Ukrainian cultural content is shown to improve Ukrainian VQA without degrading English performance, pointing toward domain adaptation as a practical path for low-resource languages.
  • The large English-to-Ukrainian degradation on translated VQA indicates that translation-based benchmarking alone understates the gap; native benchmarks like ZNO-Vision complement translated ones.
  • The code-switching and Ukrainian-dish misattribution results suggest that training-data curation must address cultural correctness, not just language fluency.
  • The benchmark's STEM-heavy, OCR-dependent question mix means it also functions as a Ukrainian OCR diagnostic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit is that per-model prompting or answer extraction would probably compress the ranking; the paper's own instruction-following observations hint that format adherence is a large hidden component of the scores.
  • The same recipe, a standardized national exam plus a small cultural benchmark, could be transplanted to other mid-resource languages, giving each language a culturally grounded rather than translated evaluation.
  • A testable extension would be to compare ZNO-Vision scores against text-only versions of the same questions; if text-only models match the multimodal results, the benchmark is measuring language knowledge more than vision grounding.
  • The cultural misattribution results could be turned into a diagnostic for fairness audits: measuring how often a model reassigns a UNESCO-recognized dish to a neighboring culture may track broader geopolitical bias in training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ZNO-Vision, a multimodal Ukrainian-language benchmark of 4,306 multiple-choice questions derived from standardized Ukrainian university entrance exams, and reports evaluations of several proprietary and open-source vision-language models on it, finding that only a handful of models beat a first-choice baseline. It also evaluates image captioning on Multi30K-UK, compares VQA 2.0 performance in English and a GPT-4o-translated Ukrainian subset, and introduces UACUISINE, a small cultural benchmark on Ukrainian dishes, including a fine-tuning experiment for PaliGemma. The central claims are that the ZNO benchmark is a valid instrument for measuring Ukrainian multimodal understanding and that current models are far from strong performance outside English.

Significance. If the ZNO-Vision benchmark is valid and the evaluations are sound, this would be a genuinely useful resource: Ukrainian is a low-resource language for multimodal evaluation, and grounding the benchmark in an external standardized exam avoids the circularity that plagues model-generated benchmarks. The paper also makes concrete falsifiable claims (e.g., systematic performance degradation on Ukrainian VQA, improvements from fine-tuning on culturally specific data). However, the manuscript's value is currently limited by three issues: the dataset and code are not released, the ZNO evaluation protocol appears to confound format-following ability with domain understanding, and the UACUISINE fine-tuning experiment has a potential train/test overlap problem. These issues bear directly on the headline empirical claims, so the significance is conditional on their resolution.

major comments (4)
  1. [§3 Evaluation Framework; §5 Results & Discussion; Table 1] The ZNO evaluation scores the last bracketed letter extracted by a regex from a single shared Ukrainian prompt, and Section 5 reports that even high-performing models frequently failed to follow the expected output format; Pixtral-12b scores 0.000 on every split and subject (Tables 1, 5, 6). This pattern suggests that the ranking may largely reflect instruction-following and format compliance rather than Ukrainian multimodal understanding, and the limitation note in Section 7 that the shared prompt 'might introduce a bias in model comparison' concedes exactly this risk for the paper's central comparison. To support the headline claim that 'only a handful of models performed above baseline,' the authors should provide per-model format-compliance rates, a manual or lenient rescoring of a representative sample, or results with multiple prompts and parsing strategies; without such evidence, the ZNO ranking is unsecured. The absence of multiple runs or confidence intervals around the ZNO accuracies, combined with temperature=1 sampling (Section 4), further weakens the comparison.
  2. [§3 Datasets & Methodology; §4 Experimental Setup] The central artifact, the ZNO-Vision benchmark, is not released: no URL, data sample, or evaluation code is provided anywhere in the paper. Because the paper's main contribution is a new benchmark and a comparative evaluation, the complete dataset with gold labels, the exact split definitions, the prompt templates, and the scoring scripts must be made publicly available (or provided as supplementary material) for the results to be reproducible and for the benchmark to be usable by others. As written, the benchmark exists only as a description.
  3. [§3 UACUISINE; §5 Fine-tuning observations; Table 4] The fine-tuning experiment is potentially circular: the UACUISINE benchmark is constructed from 20 dishes with 140 question-answer pairs (Section 3), yet the fine-tuning training set is described as 4,615 image-question-answer triplets from the UACUISINE training dataset, and no train/test split or overlap analysis is reported. If the evaluation set is drawn from the same dishes and augmented images used for training, the reported improvement from exact match 0.03 to 0.34 (Table 4) would demonstrate memorization rather than generalization. The authors must clarify the exact provenance of the training and evaluation examples and, ideally, evaluate the fine-tuned model on held-out dishes or a disjoint split.
  4. [§3 VQA 2.0; §5 Performance degradation; Table 3] The translated VQA-1000-UA set is produced entirely by GPT-4o, and the paper reports no human evaluation, back-translation checks, or inter-annotator agreement for translation quality. Since GPT-4o is itself one of the evaluated models, the reported Ukrainian-vs-English degradation in Table 3 may be inflated or distorted by translation artifacts; the Section 7 acknowledgment of this limitation is not accompanied by any diagnostic measure. The authors should either provide a translation-quality assessment (e.g., human adequacy judgments or a second translation source) or temper the claim that the degradation 'is a manifestation of cultural and linguistic bias by the models.'
minor comments (5)
  1. [Throughout] The manuscript contains several typos and informal expressions: 'Additionaly' in Section 1, 'de found' in the Table 1 caption, 'benchmank' in Section 7, 'extraxt' for 'extract,' and 'quadratic braces' for 'square brackets' in Section 3. These should be corrected for a journal submission.
  2. [Table 1 vs Appendix A] Table 1 omits models that appear in the appendix, notably Llama-3.2-90B-Vision-Instruct and Qwen2-VL-2B, without stating a selection criterion. The main results table should either include all evaluated models or explain why some are relegated to the appendix.
  3. [Figure 1 and Table 7] Figure 1 shows subject categories such as Accounting and Finance, Economics, Law, and Political Science, but Table 7 lists a different set of categories with no mapping between them. The category taxonomy should be consistent across the figure, tables, and text.
  4. [§5 Fine-tuning observations] The text says intersection match improved 'from 0% to 15%,' but Table 4 shows the highest IM is 0.30 (at 1 epoch) and the 4-epoch value is 0.15. Please correct the text to match the table, or clarify which epoch is being referenced.
  5. [References] The references to 'eie, 2024' and 'Osv, 2024' are bare access URLs in the text with no formal citation details; they should be converted to standard bibliographic entries. The GQA reference cites 'Ainslie et al., 2023' but the cited paper appears to be about multi-query transformer models, not the compositional QA dataset, so the reference list should be checked for correctness.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ZNO-Vision and VQA-UA rest on external data; observed prompt-format issues are validity threats, not circular derivation.

full rationale

ZNO-Vision is constructed from external standardized exam questions collected from the Osvita portal and is scored against a simple first-choice baseline; model outputs are not used to build or label the benchmark, so the 'only a handful of models performed above baseline' result is an independent empirical measurement rather than a fitted prediction. The translated VQA 2.0 evaluation uses a subset of an external benchmark with the original VQA accuracy protocol, and the GPT-4o translation is an input to the measurement, not a quantity derived from the evaluated models' outputs. The UACUISINE fine-tuning experiment is a standard train/test comparison: Paligemma is trained on UACUISINE training triplets and then evaluated on the UACUISINE benchmark, and the reported improvement (EM from 0.03 to 0.34) is an empirical outcome, not an identity forced by construction. The paper contains two self-citations (Paniv et al. 2024 for GPT-4o translation quality and Kiulian et al. 2024 for code-switching behavior), but both are peripheral supporting observations rather than load-bearing premises of the central claims. Section 7's concessions about shared-prompt bias and machine-translation bias, together with Section 5's documented instruction-following failures (including Pixtral's 0.000 scores), are real threats to the external validity of the model ranking, but they are not circularity: they concern whether the benchmark measures Ukrainian multimodal ability rather than format compliance, not whether the paper's derivations reduce to their own inputs. No equation or evaluation step in the paper is equivalent to its input by definition.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the correctness of the ZNO answer keys, the quality of the GPT-4o translation, and the fairness of the fixed evaluation protocol. There are no fitted free parameters in the headline results, and no new physical or conceptual entities are introduced.

assumptions (4)
  • domain assumption The ZNO answer keys from the Osvita portal are correct and the filtered single-letter questions are valid multimodal exam items.
    The benchmark's ground truth is taken from the public exam portal without independent verification (Section 3).
  • domain assumption GPT-4o's translation of VQA 2.0 into Ukrainian preserves the semantics and naturalness needed for a fair comparison with English.
    VQA-UA results and the degradation analysis in Section 5 rely on this; the authors list machine translation as a limitation in Section 7.
  • domain assumption A single fixed prompt prefix and last-bracket regex parser measure model accuracy fairly across all models.
    Section 5 reports pervasive instruction-following failures; Pixtral scored 0.000 on all ZNO splits, suggesting format sensitivity.
  • domain assumption Filtering ZNO questions to single-image, single-letter, non-matching items still yields a representative academic benchmark.
    The filtered set is heavily STEM and OCR dominated (Section 3, Table 7), so the benchmark's breadth is limited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains." pith.science (2026). https://pith.science/paper/TZBYZ73T

@misc{pith2026241114647,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TZBYZ73T}},
  note         = {Machine review of arXiv:2411.14647}
}
read the original abstract

While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from standardized university entrance examination (ZNO). The benchmark consists of over 4,300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset, translated the VQA benchmark into Ukrainian, and measured performance degradation relative to original English versions. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages.

Figures

Figures reproduced from arXiv: 2411.14647 by the authors.

Figure 1
Figure 1. Distribution of ZNO questions by year and [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 6 canonical work pages

  1. [1]

    https://testportal.gov.ua/zagalna-informatsiya-zno/ External independent evaluation

    2024. https://testportal.gov.ua/zagalna-informatsiya-zno/ External independent evaluation . Accessed: 2024-11-03

    work page 2024

  2. [2]

    https://osvita.ua/test/ Osvita.ua test portal

    2024. https://osvita.ua/test/ Osvita.ua test portal . Accessed: 2024-11-03

    work page 2024

  3. [3]

    Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2015. https://visualqa.org/evaluation.html Visual question answering (vqa) - evaluation . Accessed: 2024-11-03

    work page 2015

  4. [4]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall,...

    arXiv 2024

  5. [5]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. https://arxiv.org/abs/2305.13245 Gqa: Training generalized multi-query transformer models from multi-head checkpoints . Preprint, arXiv:2305.13245

    arXiv 2023

  6. [6]

    Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33--40

    2007

  7. [7]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 and claude with code interpreter . Accessed: 2024-11-21

    work page 2024

  8. [8]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. https://arxiv.org/abs/1505.00468 VQA: visual question answering . CoRR, abs/1505.00468

    arXiv 2015

Show all 33 references
  1. [9]

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil H...

    2024 arXiv

  2. [10]

    Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut

    Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V. Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023. https://arxiv.org/abs/2209.05401 Maxm: Towards multilingual visual question answering . Preprint, arXiv:2209.05401

    2023 arXiv

  3. [11]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    2018 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  5. [13]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2016. https://arxiv.org/abs/1612.06890 Clevr: A diagnostic dataset for compositional language and elementary visual reasoning . Preprint, arXiv:1612.06890

    2016 arXiv

  6. [14]

    Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, and Adarsh Shirawalmath. 2024. https://arxiv.org/abs/2404.09138 From bytes to borsch: Fine-tuning gemma and mistral for the ukrainian language representation . Preprint, arXiv:2404.09138

    2024 arXiv

  7. [15]

    Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. 2024. https://arxiv.org/abs/2408.08632 A survey on benchmarks of multimodal large language models . Preprint, arXiv:2...

    2024 arXiv

  8. [16]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. 2024. https://arxiv.org/abs/2305.07895 Ocrbench: On the hidden mystery of ocr in large multimodal models . Preprint, arXiv:2305.07895

    2024 arXiv

  9. [17]

    Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hendricks, Karolina Stańczak, and Aishwarya Agrawal. 2024. https://arxiv.org/abs/2407.10920 Benchmarking vision language models for cultural understanding . Preprint, arXiv:2407.10920

    2024 arXiv

  10. [18]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  11. [19]

    Yurii Paniv, Dmytro Chaplynskyi, Nikita Trynus, and Volodymyr Kyrylov. 2024. https://aclanthology.org/2024.unlp-1.6 Setting up the data printer with improved E nglish to U krainian machine translation . In Proceedings of the Third Ukrainian Natural Language Processing Workshop...

    2024

  12. [20]

    ChaeHun Park, Koanho Lee, Hyesu Lim, Jaeseok Kim, Junmo Park, Yu-Jung Heo, Du-Seong Chang, and Jaegul Choo. 2024. https://arxiv.org/abs/2406.02331 Translation deserves better: Analyzing translation artifacts in cross-lingual visual question answering . Preprint, arXiv:2406.02331

    2024 arXiv

  13. [21]

    Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych

    Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O. Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. 2022. https://arxiv.org/abs/2109.06082 xgqa: Cross-lingual visual question answering . Preprint, arXiv:2109.06082

    2022 arXiv

  14. [22]

    Matt Post. 2018. https://www.aclweb.org/anthology/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Belgium, Brussels. Association for Computational Linguistics

    2018

  15. [23]

    Mariana Romanyshyn, Oleksiy Syvokon, and Roman Kyslyi. 2024. https://aclanthology.org/2024.unlp-1.9 The UNLP 2024 shared task on fine-tuning large language models for U krainian . In Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP) @ LREC-COLING 2...

    2024

  16. [24]

    Nataliia Saichyshyna, Daniil Maksymenko, Oleksii Turuta, Andriy Yerokhin, Andrii Babii, and Olena Turuta. 2023. https://doi.org/10.18653/v1/2023.unlp-1.7 Extension M ulti30 K : Multimodal dataset for integrated vision and language research in U krainian . In Proceedings of the...

    2023 doi

  17. [25]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. https://arxiv.org/abs/2206.01718 A-okvqa: A benchmark for visual question answering using world knowledge . Preprint, arXiv:2206.01718

    2022 arXiv

  18. [26]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

    2024 arXiv

  19. [27]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12191 Qwen2-v...

    2024 arXiv

  20. [28]

    Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Chin...

    2024 arXiv

  21. [29]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. htt...

    2024 arXiv

  22. [30]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  23. [31]

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. https://arxiv.org/abs/1511.03416 Visual7w: Grounded question answering in images . Preprint, arXiv:1511.03416

    2016 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.