REVIEW 4 major objections 3 minor
Team PA-VCG's Solution for Competition on Understanding Chinese College Entrance Exam Papers in ICDAR'25
T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A document-understanding system that combines high-resolution image processing, multi-image end-to-end input, and domain-specific post-training reaches 89.6% accuracy on Chinese Gaokao paper understanding, taking first place in the…
desk verdict A thin competition report: the 89.6% leaderboard score is real evidence, but with no metric definition or method details, the scientific content is unassessable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three mechanisms carry the argument. High-resolution image processing keeps small characters and dense OCR text legible to the model. The multi-image end-to-end input strategy feeds several images of a document into the model at once, preserving cross-page layout context that single-image inputs would lose. Domain-specific post-training then fine-tunes the model on the distinctive structures found in Gaokao papers, such as multi-part questions and tables. The authors single out the post-training as the component that delivers the winning accuracy.
What would settle it
Re-run the described pipeline on a fresh set of Gaokao paper questions that were not part of the competition test set. If accuracy falls far below 89.6%, or if the ranking among entries changes, the original result would look like benchmark-specific overfitting rather than a robust understanding of exam papers.
Extended reading notes
Core claim
The central claim is that the authors' full pipeline—high-resolution image input to preserve dense text, a multi-image end-to-end strategy that lets the model see multiple pages or sections as one context, and domain-specific post-training on exam-paper material—achieves the best score in the ICDAR'25 Gaokao paper understanding competition, 89.6% accuracy. The paper presents this as evidence that a general document-understanding model, when adapted to a narrowly structured domain, can outperform other entries that do not use this type of specialized post-training. On its own terms, the result establishes the proposed combination as the current top performer on this benchmark.
Load-bearing premise
The competition's benchmark and accuracy metric genuinely measure how well a system understands exam papers, and the 89.6% score is not an artifact of overfitting to the particular test set; the abstract gives no details about the evaluation data or metric.
Editorial extensions
If this is right
- If the reported result holds, the proposed system outperforms all other entries in the ICDAR'25 Gaokao paper understanding competition.
- The strategy demonstrates that dense OCR and complex document layouts can be handled by high-resolution and multi-image inputs, without describing a separate OCR correction stage.
- Domain-specific post-training on a narrow document type is presented as an effective way to lift a general document-understanding model to a competitive lead.
- The approach gives future competition participants a concrete recipe: preserve image resolution, provide multi-page context, and post-train on the target domain.
Reading between the lines
- The abstract does not isolate the contribution of each component; it is possible that high-resolution input alone, or the multi-image strategy alone, would capture much of the gain, and the post-training matters less than suggested.
- Because success depends on post-training data from Gaokao papers, transferring the same recipe to other exam styles or document types would require collecting analogous domain data, so the result is not automatically general.
- An 89.6% accuracy leaves a non-trivial error rate; the remaining failures may concentrate in handwritten answers, rotated or noisy scans, and question formats unseen in post-training.
- If the benchmark's test set is fixed and publicly known, subsequent systems could be tuned to it, so the relative ranking may not persist without a hidden test set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript is an abstract-only report of the authors' solution to the ICDAR 2025 competition on understanding Chinese college entrance exam papers. It claims first place with an accuracy rate of 89.6%, attributed to high-resolution image processing, a multi-image end-to-end input strategy, and domain-specific post-training. No methodological details, evaluation protocol, or comparison to other methods are provided beyond this abstract.
Significance. If the reported accuracy is measured on a well-defined and representative benchmark, a first-place finish in an ICDAR competition is meaningful evidence of competitive performance on a real-world document understanding task. The reliance on an external leaderboard score is a strength in terms of avoiding circular evaluation. However, the scientific significance is currently limited because the abstract does not define the metric or provide an analysis of what the system actually learns, making the central claim an uninterpreted factoid rather than a demonstrated contribution.
major comments (4)
- [Abstract (sole content)] The submitted manuscript consists solely of the abstract; it contains no description of the model architecture, training data, post-training procedure, evaluation protocol, or comparison with prior work. A refereed journal submission must contain the full technical details, and without them the central claim of first place cannot be independently assessed or reproduced.
- [Abstract ("accuracy rate")] The central claim of "accuracy rate of 89.6%" is not accompanied by a definition of the metric. The ICDAR competition may involve multiple sub-tasks such as OCR, layout analysis, and question answering; if the accuracy is an exact-match of extracted answer strings, it may reflect OCR and template-following capability rather than semantic understanding of exam content. The paper must specify the task definition, the metric, and the test set size, and ideally break down performance per sub-task.
- [Abstract ("domain-specific post-training strategies")] The abstract attributes the performance to "domain-specific post-training strategies" but gives no details about the base model, the post-training data, the training objective, or the hyperparameters. Since the claimed novelty lies in this post-training approach, omitting these details makes the approach neither reproducible nor falsifiable. At minimum, an ablation or an analysis of what the post-training changes should be reported.
- [Abstract ("understanding")] The paper claims gains in "understanding" but the described technical elements (high-resolution image processing, dense OCR extraction, multi-image input) are primarily visual and layout-related. The authors should justify that the benchmark metric actually measures understanding, e.g., by reporting qualitative examples or a per-category breakdown that separates OCR/layout errors from reasoning errors. Without this, the claim that the method "understands" exam papers is not supported.
minor comments (3)
- [Title vs Abstract] The team name is "PA-VCG" in the title but "PA-VGG" in the abstract; please unify the spelling.
- [Abstract] The notation "ICDAR'25" should be typeset consistently (e.g., "ICDAR 2025").
- [Abstract] The phrase "our post-training approach" appears without first introducing any baseline; consider clarifying which model or system the post-training is applied to.
Circularity Check
No circularity found: the claimed accuracy is benchmarked against an external ICDAR competition.
full rationale
The paper is an abstract-only competition report. Its central claim is a first-place accuracy rate of 89.6% on the ICDAR'25 Competition on Understanding Chinese College Entrance Exam Papers. The result is evaluated against an external benchmark with an outside test set, not derived from the claim itself. There are no equations, fitted parameters, or self-citations in the provided text from which a circular reduction could be exhibited. The skeptical concern about the undefined metric is a validity or interpretability issue, not a circularity issue: the leaderboard result is an external measurement rather than an input to the method. Per the hard rules, lack of metric definition and potential overfitting concerns are correctness risks, not evidence of circularity. Therefore the appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
assumptions (1)
- domain assumption The competition evaluation set and metric are reliable and representative of understanding Chinese exam papers.
Cite this review
Pith. "Pith review of Team PA-VCG's Solution for Competition on Understanding Chinese College Entrance Exam Papers in ICDAR'25." pith.science (2026). https://pith.science/paper/GWW2UTDO
@misc{pith2026250800834,
author = {Pith},
title = {Pith review of: Team PA-VCG's Solution for Competition on Understanding Chinese College Entrance Exam Papers in ICDAR'25},
year = {2026},
howpublished = {\url{https://pith.science/paper/GWW2UTDO}},
note = {Machine review of arXiv:2508.00834}
}
read the original abstract
This report presents Team PA-VGG's solution for the ICDAR'25 Competition on Understanding Chinese College Entrance Exam Papers. In addition to leveraging high-resolution image processing and a multi-image end-to-end input strategy to address the challenges of dense OCR extraction and complex document layouts in Gaokao papers, our approach introduces domain-specific post-training strategies. Experimental results demonstrate that our post-training approach achieves the most outstanding performance, securing first place with an accuracy rate of 89.6%.
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.