Pith. sign in

REVIEW 3 major objections 4 minor 21 references

KRETA, a 2,577-item native Korean VQA benchmark, separates reading from reasoning and finds open vision-language models collapse on reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

KRETA, a 2,577-item Korean text-rich VQA benchmark, shows vision-language models recognize Korean text well but lag in multi-step reasoning, especially in open-source models.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection KRETA is a genuinely useful new Korean text-rich VQA benchmark with a sensible System 1/System 2 split; the main open question is whether the LLM-built questions are valid without a human performance baseline. the 3 major comments →

arxiv 2508.19944 v2 pith:OKVBWV5G submitted 2025-08-27 cs.CV cs.CL

KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts

Taebaek Hwang , Minseo Kim , Gisang Lee , Seonuk Kim , Hyunjun Eun This is my paper
classification cs.CV cs.CL
keywords Korean text-rich VQAvision-language modelsSystem 1 / System 2 reasoningmultilingual benchmarkmultiple-choice QAsemi-automated data pipelinelow-resource language evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KRETA gives Korean text-rich visual question answering a proper measuring instrument. It consists of 2,577 questions built from native Korean images and text, split into System 1 items that require reading text and System 2 items that require multi-step reasoning, domain knowledge, or visual math. The paper finds that current vision-language models, especially open-source ones, handle basic Korean text recognition well but collapse on System 2 reasoning; Qwen2.5-VL-7B falls from 94.5% to 36.1%. The paper attributes this to thin Korean-contextual and domain-specific training data. If the benchmark is valid, it gives researchers a way to separate reading ability from reasoning ability in Korean, and its semi-automated generation pipeline offers a template for similar benchmarks in other low-resource languages.

Core claim

The paper claims that KRETA, built only from native Korean imagery and questions rather than translated English content, is the first large-scale Korean text-rich VQA benchmark capable of evaluating both basic text recognition and advanced reasoning. Evaluation results show a 'reasoning bottleneck': closed models average about 70.5% on System 2 questions, while open-source models score as low as 29-49%. The largest drops occur in culturally rich domains like CSAT History and in cluttered image types like banners and store signs. The paper argues this gap is not mainly an OCR failure, because System 1 recognition accuracy is high, but rather a deficiency in Korean-contextual, domain-specific

What carries the argument

The central object is KRETA itself: 2,577 multiple-choice QA pairs grounded in native Korean images, organized by a dual-level System 1 versus System 2 split and by a 15-domain, 26-image-type taxonomy. The argument is carried by a semi-automated pipeline: two VLMs independently decompose each image into structured captions capturing layout, text, and text-visual links; LLMs generate QA candidates; two VLMs score candidates on a seven-metric protocol; an LLM synthesizes hard distractor options; and human review refines the survivors. The System 1/System 2 distinction makes the benchmark diagnostic rather than just a leaderboard, because it lets the authors attribute performance drops to reaso

Load-bearing premise

The benchmark's validity rests on the assumption that the automatically generated QA pairs, chosen by model judges and only lightly refined by humans, are correct, unambiguous, and not biased toward the generating models; the paper provides no human performance baseline or inter-annotator agreement to back this assumption.

What would settle it

Take 200 randomly selected KRETA questions, give the image and options to native Korean speakers without exposing the gold answer, and compare their choices with the released key; also log how often a System 2 question is answerable from text alone. Low agreement, or a sizable share of text-only-answerable items, would show the measured reasoning gap is an artifact of question construction rather than model reasoning.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • KRETA can serve as a reusable testbed for Korean text-rich VQA across 15 domains and 26 image types, enabling targeted diagnosis of where a model's reading and reasoning skills diverge.
  • Because model rankings change sharply between System 1 and System 2, evaluations should report recognition and reasoning separately rather than as one aggregate score.
  • Open-source VLMs need targeted Korean-contextual and domain-specific training, especially for CSAT Science and History and for complex real-world layouts like banners and store signs.
  • Chain-of-Thought prompting is not universally helpful: it improves capable closed models but degrades small open models, so prompt effects must be part of deployment decisions.
  • The semi-automated pipeline can be adapted to other low-resource languages, provided prompts are written natively in the target language rather than translated from English.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the pipeline runs on structured captions rather than raw OCR, it could plausibly be lifted to other undersourced languages, though the paper only gestures at this transfer.
  • The System 2 category bundles several distinct reasoning types - multi-step deduction, math, domain knowledge - so fine-grained labels would reveal whether open models fail uniformly or mostly on knowledge-heavy items; the paper's own limitations section concedes this.
  • The prompt-language effect (Korean CoT boosting Qwen2.5-VL-7B while English CoT hurts it) suggests cross-lingual benchmark comparisons should control evaluation-language choice, otherwise measured gaps may partly reflect prompt language rather than model capability.
  • If the benchmark is valid, the 40-plus-point closed-open System 2 gaps on culturally dense domains imply that simply scaling a multilingual model without preserving low-resource language data share can dilute Korean-specific ability, a risk the paper raises in its model-size analysis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces KRETA, a Korean text-rich VQA benchmark containing 2,577 multiple-choice QA pairs over native Korean images, organized into 15 domains and 26 image types and split into System 1 (basic text recognition, 1,426 items) and System 2 (advanced reasoning, 1,151 items). QA items are produced semi-automatically: two VLMs decompose images into structured captions, LLMs generate candidates, LLM judges score them on up to seven metrics, hard negatives are added, and human refinement is applied. The authors evaluate 18 closed- and open-source VLMs, reporting that closed models reach roughly 80--85% overall accuracy while open models lag, and that System 2 causes large drops (e.g., Qwen2.5-VL-7B from 94.5% to 36.1%). They also analyze results by domain, image type, model size, and chain-of-thought prompting, and release the code, data, prompts, and a leaderboard.

Significance. KRETA addresses a real gap: it is among the largest native Korean text-rich VQA datasets, with a useful dual-level reasoning split and a domain taxonomy tied to KSIC. The CSAT subsets are grounded in official examination materials, providing a valuable externally anchored component. A further strength is the release of the pipeline, prompts, and evaluation code, which supports reproducibility and adaptation to other low-resource languages. The empirical finding that open-source models collapse on System 2 items is interesting and actionable. However, the paper's central claim is that KRETA constitutes a valid measurement instrument, and the current evidence for that claim is incomplete: the benchmark's correctness and labeling rest on unquantified LLM generation/judging plus an unspecified human refinement step, and the generator/judge models also appear in the evaluation pool. These issues must be addressed before the numerical conclusions can be fully trusted.

major comments (3)
  1. The validity of KRETA as a measurement instrument is not established. Step 2 uses GPT-4o-mini, Gemini-2.0-flash, and o1-mini to propose QA pairs; Step 3 has GPT-4o-mini and Gemini-2.0-flash score the candidates; the human refinement step is described but no quantitative results are reported. There is no human accuracy baseline, no inter-annotator agreement, no per-item rejection rate, and no check of text-only answerability or ambiguity. Consequently, the reported System 1/System 2 gap (e.g., Qwen2.5-VL-7B 94.5 vs. 36.1) could in part be an artifact of the generation pipeline rather than a measurement of VLM reasoning. Moreover, the same models that generate and judge the data (GPT-4o-mini, Gemini-2.0-flash) are later evaluated on it in Table 2, risking inflated scores due to familiarity with their own item style. Please add human validation statistics (accuracy on a sample, agreement, e
  2. Several cross-domain and image-type conclusions are drawn from very small samples without uncertainty quantification. For example, CSAT History has 60 items (Table 3), and several image types in Table 4 are near the 50-item cutoff; a 10--15 percentage point gap is close to the binomial standard error for such cells. The statement that GPT-4o 'excels' in CSAT History (93.3%) versus other models should be supported by confidence intervals or significance tests. The text should temper conclusions about specific domains and image types where the sample sizes are too small to support reliable ranking.
  3. The System 1/System 2 construct validity is under-verified. The labels are assigned by the same generation prompts and LLM judges rather than by an independent protocol, and no evidence is provided that System 2 items actually require multi-step inference or that System 1 items are purely recognition-based. The Limitation section candidly notes that System 2 conflates several reasoning types, but that admission does not substitute for a validation study. Please report human agreement on System 1 vs. System 2 labels, or an analysis of the number of inference steps and answerability-without-image for a random sample of items.
minor comments (4)
  1. The reference 'Kim et al., 2025' lists an incomplete arXiv identifier (arXiv:2505.XXXXX). Please update before publication.
  2. The chain-of-thought results are described with point differences (e.g., +3.7, -7.7) but no error bars or per-model tables are provided. Please include confidence intervals or a table with exact scores so the reader can assess the reliability of these differences.
  3. The table as rendered in the manuscript has garbled numeric spacing (e.g., the GPT-4o row), which makes verification difficult. Please ensure the final typeset version is readable and aligned.
  4. The comparison between CSAT Science (478 items) and CSAT History (60 items) should be conditioned on the large sample-size difference; statements such as 'GPT-4o excels in CSAT History' should be softened or qualified.

Circularity Check

0 steps flagged

No significant circularity: benchmark construction is not a fitted prediction chain, and evaluation results are measurements on a fixed dataset.

full rationale

I found no load-bearing circular step in the paper's derivation chain. The core contribution is a dataset/benchmark, not a predictive model fitted to data. The semi-automated pipeline in Section 3.3 uses GPT-4o-mini and Gemini-2.0-flash to generate and score QA candidates, and those same models later appear in the evaluation table (Table 2). This overlap is a legitimate validity concern—generator-judge models may be advantaged by familiarity with their own question style—but it is not a formal circularity: no parameter is fitted to the reported accuracies, the final QA pairs undergo human refinement (Section 3.3, 'Human Annotation Refinement'), and the reported System 1 vs. System 2 accuracies are observations on a fixed benchmark, not predictions derived from the generating models. The System 1/System 2 distinction is a design choice informed by Kahneman's framework and by the pipeline's prompt design, not a result that reduces to its own definition. The limitations section also candidly notes that System 2 conflates multiple reasoning types, further indicating the split is an analytic taxonomy rather than a forced mathematical consequence. The absence of a human performance baseline or inter-annotator agreement is a missing-evidence concern about benchmark validity, not a circularity concern under the strict definition used here. No self-citations are load-bearing, no uniqueness theorem is imported from the authors' prior work, and no known empirical result is merely renamed. Overall, the paper's claims are self-contained as a benchmark construction and evaluation study.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claim rests on the validity of the KRETA instrument itself, which is constructed via LLM generation and selection, with human refinement described but not quantified. No free parameters are fitted, and no new theoretical entities are introduced.

axioms (4)
  • domain assumption LLM-generated QA pairs, after LLM-judge selection and human refinement, are correct and unambiguous.
    Central to benchmark validity; no human performance baseline or inter-annotator agreement is reported (Section 3.3, Limitations).
  • domain assumption The System 1/System 2 labeling produced by generation prompts reflects distinct recognition vs. reasoning levels.
    The paper itself concedes System 2 conflates sequential deduction, external knowledge, and cross-referential analysis (Limitations).
  • domain assumption Multiple-choice accuracy reflects VLM ability, not guessing or positional bias.
    A 25% chance baseline is not provided; only raw accuracies on the 'Answer: LETTER' protocol are reported (Section 4).
  • domain assumption Image filtering thresholds (shortest side >=384 px, 10-1000 Korean characters) do not bias benchmark difficulty.
    The filters may exclude hard or low-text images; their effect on the System 1/System 2 gap is unanalyzed (Section 3.3 Step 1).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts." pith.science (2026). https://pith.science/paper/OKVBWV5G

@misc{pith2026250819944,
  author       = {Pith},
  title        = {Pith review of: KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKVBWV5G}},
  note         = {Machine review of arXiv:2508.19944}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding and reasoning over text within visual contexts poses a significant challenge for Vision-Language Models (VLMs), given the complexity and diversity of real-world scenarios. To address this challenge, text-rich Visual Question Answering (VQA) datasets and benchmarks have emerged for high-resource languages like English. However, a critical gap persists for low-resource languages such as Korean, where the lack of comprehensive benchmarks hinders robust model evaluation and comparison. To bridge this gap, we introduce KRETA, a benchmark for Korean Reading and rEasoning in Text-rich VQA Attuned to diverse visual contexts. KRETA facilitates an in-depth evaluation of both visual text understanding and reasoning capabilities, while also supporting a multifaceted assessment across 15 domains and 26 image types. Additionally, we introduce a semi-automated VQA generation pipeline specifically optimized for text-rich settings, leveraging refined stepwise image decomposition and a rigorous seven-metric evaluation protocol to ensure data quality. While KRETA is tailored for Korean, we hope our adaptable and extensible pipeline will facilitate the development of similar benchmarks in other languages, thereby accelerating multilingual VLM research. The code and dataset for KRETA are available at https://github.com/tabtoyou/KRETA.

Figures

Figures reproduced from arXiv: 2508.19944 by Gisang Lee, Hyunjun Eun, Minseo Kim, Seonuk Kim, Taebaek Hwang.

Figure 1
Figure 1. Figure 1: (a) Distribution of samples across 15 domains (inner ring) and 26 image types (outer ring). Dark green [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Examples from KRETA, showcasing diverse domains and image types categorized under System 1 and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of open-source and closed mod [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of two closed and four open￾source models of varying sizes on KRETA. The figure shows performance differences across three prompts: Baseline, Chain-of-Thought in English and Korean. models’ superior ability to integrate background knowledge and cultural context ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An overview of the semi-automated VQA generation pipeline with prompts. Each step involves data [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages · 4 internal anchors

  1. [1]

    arXiv preprint arXiv:2407.06023. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024a. MMMU: A massive multi-discipline multimodal under...

    Pith/arXiv arXiv

  2. [2]

    Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration

    Evaluat- ing visual and cultural interpretation: The k-viscuit benchmark with human-vlm collaboration. arXiv preprint arXiv:2406.16469. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [4]

    target-language in- structions for multilingual LLMs

    A fair comparison with- out translationese: English vs. target-language in- structions for multilingual LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (V olume 2: Short Papers), pages 649–670, Albuquerque, New Mexico. Association for Computa...

    work page 2025

  4. [5]

    QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

    QGEval: A benchmark for question generation evaluation. arXiv preprint arXiv:2406.05707. Google DeepMind

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [6]

    arXiv preprint arXiv:2501.05444

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444. Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen

    Pith/arXiv arXiv

  6. [8]

    arXiv preprint arXiv:2411.19103

    V ARCO-VISION: Expanding frontiers in korean vision-language models. arXiv preprint arXiv:2411.19103. Daniel Kahneman

    Pith/arXiv arXiv

  7. [9]

    KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language

    Koffvqa: An objectively evaluated free-form vqa benchmark for large vision-language models in the korean language. arXiv preprint arXiv:2503.23730. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [10]

    arXiv preprint arXiv:2408.03326

    LLaV A- OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual Instruction Tuning. In Advances in Neural Information Processing Systems (NeurIPS). Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Ji- aqi Wang, Conghui He, Ziwei Liu, ...

    Pith/arXiv arXiv

  9. [11]

    arXiv preprint arXiv:2405.20797

    OVIS: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque

    Pith/arXiv arXiv

  10. [12]

    arXiv preprint arXiv:2203.10244

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244. Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har

    Pith/arXiv arXiv

  11. [14]

    In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2497–

    xGQA: Cross-lingual visual ques- tion answering. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2497–

    work page 2022

  12. [15]

    arXiv preprint arXiv:2406.02539

    Parrot: Mul- tilingual visual instruction tuning. arXiv preprint arXiv:2406.02539. Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et al. 2024a. TextSquare: Scaling up text-centric visual instruction tuning. arXiv preprint arXiv:2404.12803. Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, C...

    Pith/arXiv arXiv

  13. [16]

    MUST-VQA: MUltilingual Scene-text VQA

    MUST- VQA: Multilingual scene-text vqa. arXiv preprint arXiv:2209.06730. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [17]

    arXiv preprint arXiv:2409.12191

    Qwen2-VL: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al

    Pith/arXiv arXiv

  15. [18]

    arXiv preprint arXiv:2412.10302

    DeepSeek-VL2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al

    Pith/arXiv arXiv

  16. [19]

    arXiv preprint arXiv:2408.01800

    MiniCPM-V: A GPT- 4V level MLLM on your phone. arXiv preprint arXiv:2408.01800. Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov

    Pith/arXiv arXiv

  17. [21]

    For readability, we omit the full prompt text here; the exact templates and configurations are provided in the repository. A.2 Impact of Prompt Language on VQA Generation We observed that directly translating prompts into English while still requesting Korean outputs con- sistently degraded QA quality compared to us- ing native Korean prompts. We hypothes...

    work page 2025

  18. [2022]

    arXiv preprint arXiv:2209.08199

    ScreenQA: Large- scale question-answer pairs over mobile app screen- shots. arXiv preprint arXiv:2209.08199. Jeongho Ju, Daeyoung Kim, SunYoung Park, and Youngjune Kim

    Pith/arXiv arXiv

  19. [2023]

    In Proceedings of ACL

    Crosslingual general- ization through multitask finetuning. In Proceedings of ACL. OpenAI. 2024a. GPT-4o main page. https:// openai.com/index/hello-gpt-4o. Gpt-4o-2024- 11-20 version [Multimodal Large Language Model]. OpenAI. 2024b. o1-mini: Advancing cost-efficient rea- soning. Product & technical overview. Version: 2024- 09-12. Jonas Pfeiffer, Gregor Ge...

    work page 2024

  20. [2024]

    arXiv preprint arXiv:2404.14219

    Phi-3 technical report: A highly ca- pable language model locally on your phone. arXiv preprint arXiv:2404.14219. Anthropic

    Pith/arXiv arXiv

  21. [2025]

    arXiv preprint arXiv:2502.13923

    Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. 2024a. MJ-Bench: Is your multimodal reward model really a good judge for text-to-image generation? arXiv preprint arXiv:2407.04842. Zhe Chen, Weiyun Wang, ...

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.