REVIEW 3 major objections 4 minor 8 cited by
The paper claims that a fully automated, distillation-free two-stage pipeline trains a 3B vision-language model for document conversion that outperforms larger public and proprietary models.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
POINTS-Reader uses a two-stage synthetic-data warm-up plus iterative self-improvement with rule-based filtering to train a 3B vision-language model for document conversion, outperforming larger models on OmniDocBench and Fox.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection The self-improvement pipeline is real and the ablations are solid, but the SOTA claim rests on a comparison table that mixes evaluation protocols, so the headline margins over Mistral OCR and OLMOCR are not yet established. the 3 major comments →
POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim is that a closed loop of self-annotation and cheap deterministic filtering can adapt a document-conversion model to real-world data without any teacher model. The key evidence is the final 3B model, POINTS-Reader: 0.259 overall on OmniDocBench (lower is better) against 0.390 for Qwen2.5-VL-3B, 0.331 for Qwen2.5-VL-7B, 0.326 for OLMOCR, and 0.268 for Mistral OCR. On the Fox English benchmark it reaches 0.023 normalized edit distance, ahead of all listed end-to-end models. The authors also report that the model outperforms a student trained by direct distillation from Qwen2.5-VL-72B, showing that the bottleneck in this setting is data quality rather than model capacit
What carries the argument
The engine of the method is the iterative self-improvement stage, powered by three rule-based filters. Plain text is filtered by an F1 score against PaddleOCR, tables are checked for consistency of cell counts across rows and columns, and mathematical formulas are checked only for LaTeX syntactic validity. These filters deliberately do not judge semantic content: they reject hallucinations, broken table structures, and malformed formulas without needing a learned teacher. A warm-up stage on synthetic pages rendered from HTML templates gives the model a strong initial grasp of the unified output formats, and each self-improvement iteration re-initializes from the pretrained backbone rather th
Load-bearing premise
The whole pipeline assumes PaddleOCR's output is a trustworthy reference for what the page actually says; if PaddleOCR systematically misreads certain elements, those misreadings become the training target.
What would settle it
Take pages where PaddleOCR is known to be wrong—formulas, footnotes, headers, unusual fonts—and compare POINTS-Reader's output to human transcription. If the model reproduces PaddleOCR's errors more than a version trained without the F1 filter does, then the measured improvement is partly alignment to the reference rather than better document reading. A simpler check: score POINTS-Reader against OmniDocBench's human-based ground truth and see if the text-metric gap over Qwen2.5-VL-7B persists.
If this is right
- Teacher-free training removes the ceiling imposed by a teacher's own errors: a student no longer inherits the teacher's hallucinations, missed tables, or wrong table structures.
- A 3B model reaches near-72B accuracy on OmniDocBench overall and beats 72B on tables, so compact end-to-end converters are viable for deployment.
- Because the filters ignore semantic correctness of tables and formulas, even cheap structural checks (cell-count consistency, LaTeX parse) can drive substantial gains.
- The final iteration yields about 1.1M real-world image-text pairs with HTML tables and no distillation, a reusable resource for further training.
- Synthetic data alone plateaus around 800k samples; the real-world self-improvement loop is what pushes performance further.
Where Pith is reading between the lines
- Because the plain-text filter uses PaddleOCR as the reference, the model is effectively trained to agree with PaddleOCR; if PaddleOCR has systematic errors on headers, footnotes, formulas, or unusual fonts, those errors are likely baked into the training signal. A human-verified test set would show how much of the gain is genuine reading versus alignment to this reference.
- The table and formula filters validate only structure and syntax, so the model may produce tables and formulas that look right but are semantically wrong; the reported improvements on those metrics may partly reflect learning formatting conventions, not content understanding.
- The same recipe—synthetic warm-up plus self-generated data filtered by deterministic checks—could transfer to other structured extraction tasks, such as charts, forms, or slides, where semantic annotation is hard but structural validity is easy to check.
- Since the final data is heavily plain-text (about 90 percent), further gains may depend on diversifying rare layouts and elements rather than scaling volume or rebalancing sample types.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a fully automated, distillation-free two-stage framework for document conversion. The Uniform Format Warm-up Stage (UWS) standardizes plain text, tables, and formulas into Markdown/HTML/LaTeX, uses an LLM to synthesize text and renders it through HTML templates, and fine-tunes POINTS-1.5. The Iterative Self-improvement Stage (ISS) runs the current model on DocMatix, filters outputs using PaddleOCR-based F1 for plain text, structural checks for tables, and LaTeX syntax checks for formulas, then retrains, repeating over iterations. The resulting 3B POINTS-Reader is reported to achieve 0.259 overall on OmniDocBench (en), outperforming Qwen2.5-VL-3B/7B, OLMOCR, and Mistral OCR, and approaching Qwen2.5-VL-72B. Ablations show monotonic gains from each data type and filter.
Significance. If the comparative numbers hold under a common protocol, the paper makes a meaningful contribution: it shows that a compact document-conversion model can be trained without teacher-model distillation, using synthetic warm-up plus iterative self-improvement with cheap rule-based filters. The paper ships a public model and provides consistent internal ablations, a scaling study, an analysis of the self-improvement loop, and an additional comparison against a Qwen2.5-VL-72B distillation baseline; these are strengths. The main barrier to accepting the headline claim is external: the state-of-the-art comparison in Table 5 mixes evaluation protocols, and the text filter's reference signal comes from a single traditional OCR engine, so the reported margins may be protocol artifacts or partly encode PaddleOCR's errors.
major comments (3)
- [§3.3, Table 5] The central claim that POINTS-Reader surpasses Mistral OCR and OLMOCR is not established because the baselines were not evaluated under a common protocol. The table footnote states that only the Qwen2.5-VL series is re-run with the same settings; all other methods use metrics from their original papers or subsequent works. OmniDocBench scores are sensitive to the prompt template, output schema (POINTS-Reader emits unified HTML/LaTeX while OLMOCR/Mistral emit Markdown), token limits, and normalization scripts. The reported margins (0.259 vs. 0.326 OLMOCR, vs. 0.268 Mistral OCR) are small enough that protocol differences could reverse them. Please provide a common-harness rerun for the open baselines, or temper the SOTA claim.
- [§2.2, Eqs. (1)–(3)] The plain-text filter uses PaddleOCR as the reference for F1, so any systematic OCR error (formulas, page numbers, headers, footnotes, unusual fonts) is treated as an error in the model's output. In the ISS loop, this can select annotations that agree with PaddleOCR rather than with ground-truth page content. The paper's own F1-threshold ablation (Table 4) admits traditional OCR 'cannot recognize formulas' and that predictions omitting minor elements such as headers or page numbers may still be useful. This makes the filtering criterion misaligned with the stated goal of high-quality data. Please add a human or multi-OCR validation sample across iterations, and quantify how often the filter removes correct predictions or retains wrong ones.
- [§3.2.2, Tables 2 and 4] The F1 threshold (0.90) and the aspect-ratio range [(2/5,5/2)] are selected by sweeping on the OmniDocBench test set. Since these hyperparameters are tuned on the benchmark used for the final headline numbers, the reported scores are an upper envelope. The risk is not that the ablations are uninformative, but that the specific choices may be overfit to OmniDocBench's English layout distribution. Please report the chosen thresholds from a held-out validation split, or show that the conclusions are stable across thresholds.
minor comments (4)
- [Throughout] Inconsistent naming: 'DocMatrix' appears in §2.2 and §3.2.2 while DocMatix is used elsewhere. Also typos: 'Instuct' (model training), 'PONTS-Reader' (Appendix A.1), 'KaTex' reference formatting.
- [§3.3] The text says 'we evaluate all models on four benchmarks,' but Table 5 lists only OmniDocBench and Fox. Clarify whether 'four benchmarks' refers to four evaluation suites or to the sub-metrics within OmniDocBench.
- [Figure 6] Caption (c) repeats 'the number of retained samples containing tables increases after filtering' from (b); presumably one should say 'formulas'.
- [Appendix A.1] Figure references appear swapped: the paragraph on loading weights refers to Figure 8, but the relevant figure is numbered 7; the later paragraph on UWS data inclusion refers to the figure below, numbered 8.
Circularity Check
Minor local circularity in the PaddleOCR-F1 quality metric; the central claims rest on independent OmniDocBench and Fox benchmarks.
specific steps
-
fitted input called prediction
[Section 2.2, 'Filtering Plain Text' and the paragraph introducing Figure 5]
"Following the approach of CCOCR (Yang et al., 2024b), we employ the F1-score to filter plain text... Samples with F1-scores below a threshold (e.g., 0.9) are discarded. ... To quantitatively assess this trend, we calculate the F1-score between the model's predictions on DocMatrix images and the results produced by PaddleOCR, averaging these scores across all images (Figure 5). The results demonstrate that, as the number of iterations increases, the model's predictions become increasingly aligned with the target outputs."
The plain-text filter accepts exactly those model outputs whose F1 against PaddleOCR exceeds a threshold, and the reported Figure 5 'data quality' metric is the same F1 against the same PaddleOCR transcripts. Retraining on the accepted subset therefore measures convergence to the filter's own reference: the statistic is selected for by the filter, so its rise is not independent evidence of improved reading ability. The paper does also provide independent OmniDocBench ablations, so this circularity is local rather than load-bearing for the main claim.
full rationale
Most of the derivation chain is self-contained against external benchmarks. The warm-up stage uses LLM-generated synthetic text with rule-based table/formula checks, and the iterative stage applies PaddleOCR-based F1 filtering, table-cell consistency checks, and LaTeX syntax checks before retraining. OmniDocBench and Fox provide independent ground truth, and the ablation tables show each added component improves those benchmarks. The self-improvement loop is not circular by construction: the model is trained on its own filtered outputs, but retention is gated by external fixed references rather than by the benchmark being reported. One local circularity exists: Figure 5 presents the model's F1 against PaddleOCR as evidence of improved generated-data quality, but this is the same statistic used to select samples in the plain-text filter. That trend is a self-fulfilling consequence of selection on that statistic and should not be read as independent validation. The Table 5 footnote also admits that non-Qwen baselines use metrics from their original papers rather than a common evaluation harness; this is a comparability limitation, not a circularity, and it does not affect the circularity score. Overall, the central claim is supported by independent evaluations, so the score is low.
Axiom & Free-Parameter Ledger
free parameters (2)
- F1-score threshold for text filtering =
0.9
- Aspect ratio range for synthetic data filtering =
width/height in (2/5, 5/2)
axioms (6)
- domain assumption PaddleOCR provides a sufficiently accurate reference for filtering plain text on real documents.
- domain assumption Table structural validity (consistent cell counts) correlates with table recognition quality.
- domain assumption LaTeX syntax correctness is a useful proxy for formula extraction quality.
- domain assumption Self-improvement on filtered pseudo-labels does not cause model collapse or accumulation of systematic errors.
- domain assumption DocMatix is representative of real-world documents for the target distribution.
- domain assumption Text generated by Qwen2.5-3B-Instruct and rendered via HTML templates captures the diversity of real document layouts.
Cite this review
Pith. "Pith review of POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion." pith.science (2026). https://pith.science/paper/X73FDBX6
@misc{pith2026250901215,
author = {Pith},
title = {Pith review of: POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion},
year = {2026},
howpublished = {\url{https://pith.science/paper/X73FDBX6}},
note = {Machine review of arXiv:2509.01215}
}
read the original abstract
High-quality labeled data is essential for training accurate document conversion models, particularly in domains with complex formats such as tables, formulas, and multi-column text. However, manual annotation is both costly and time-consuming, while automatic labeling using existing models often lacks accuracy in handling such challenging scenarios. Consequently, training student models by distilling outputs from teacher models can significantly limit their performance in real-world applications. In this paper, we propose a fully automated, distillation-free framework comprising two stages for constructing high-quality document extraction datasets and models capable of handling diverse document formats and layouts. In the first stage, we introduce a method for generating large-scale, diverse synthetic data, which enables a model to extract key elements in a unified format with strong initial performance. In the second stage, we present a self-improvement approach that further adapts the model, initially trained on synthetic data, to real-world documents. Specifically, we first use the fine-tuned model to annotate real documents, then apply a suite of filtering strategies to verify annotation quality, and finally retrain the model on the verified dataset. By iteratively repeating this process, we progressively enhance both the model's conversion capabilities and the quality of the generated data. We train a public POINTS-1.5 model to obtain POINTS-Reader, which surpasses many existing public and proprietary models of comparable or larger size. Our model is available at https://github.com/Tencent/POINTS-Reader.
Figures
Forward citations
Cited by 8 Pith papers
-
DocAtlas: Multilingual Document Understanding Across 80+ Languages
DocAtlas creates multilingual document datasets across 82 languages and shows DPO with rendered ground truth improves model accuracy by 1.7-1.9% without degrading base-language performance, unlike supervised fine-tuning.
-
DocAtlas: Multilingual Document Understanding Across 80+ Languages
DocAtlas introduces model-free rendering pipelines to create DocTag-annotated datasets across 82 languages and shows DPO adaptation improves multilingual performance without base-language degradation.
-
Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing
PaddleOCR-VL uses a Valid Region Focus Module to select key visual tokens and a 0.9B model for guided recognition, delivering SOTA document parsing with far fewer tokens and parameters.
-
Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training
A realistic scene synthesis strategy and document-aware training recipe enable a 1B-parameter MLLM to achieve superior accuracy and robustness in end-to-end parsing of real-world captured documents.
-
HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding
A training-free, two-stage speculative decoding scheme accelerates VLM document parsers by ~2.8x end-to-end (up to 7x) while keeping parsing accuracy essentially unchanged.
-
MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing
MinerU2.5 uses a two-stage decoupled vision-language architecture to achieve state-of-the-art document parsing accuracy with lower computational overhead than existing general and domain-specific models.
-
PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing
PaddleOCR-VL-1.5 is a 0.9B VLM achieving 94.5% SOTA accuracy on OmniDocBench v1.5, with added robustness to physical distortions and support for seal recognition plus text spotting.
-
PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training
PaddleOCR-VL-1.6 improves on PaddleOCR-VL-1.5 via region-aware data optimization and progressive post-training to reach 96.33% on OmniDocBench v1.6.
Reference graph
Works this paper leans on
-
[1]
Choose "TOPIC" as the topic with a word count of approximately 300-500 words
-
[2]
The language and formatting style of the text should be chosen from the provided styles below
-
[3]
Do not include any tables or mathematical formulas in the text
-
[4]
You may choose to use some of the following MarkDown syntax elements in your writing: - Different levels of headings - Bold - Italics - Bold italics - Underline - Superscript - Subscript - Lists - Unordered lists - Ordered lists
-
[5]
The overall style and organization of the text should be more varied, **avoid always adding conclusions or summaries at the end**
-
[6]
The content of the text does not have to be complete; it can be an excerpt
-
[7]
Only return the generated text content without any additional explanations, such as descriptions before or after the text
-
[8]
Do not mention the selected language, or formatting style in the text
-
[9]
Please provide the response in English
-
[10]
Random seed is SEED Available text languages and formatting styles: Exam paper, slides, academic paper, book, textbook, magazine, notes, newspaper, financial report Prompt for Formulas Please choose one of the following topics and write a MarkDown text with the following requirements with a random seed:
-
[11]
Do not indicate the chosen topic, text language, and format style in the generated text
-
[13]
Choose "TOPIC" as the topic and create a text of about 300-400 words
-
[14]
The language and format style of the text should be chosen from the given styles
-
[15]
- Multiline formula styles, such as equation&split, align, gather, alignat, etc
You can insert some LaTeX formulas, choosing from the following formula styles: - Matrix styles, such as matrix, array, pmatrix, bmatrix, vmatrix, Vmatrix, Bmatrix, cases, rcases, smallmatrix, subarray, etc. - Multiline formula styles, such as equation&split, align, gather, alignat, etc. - Regular mathematical formula styles, such as frac, sum, etc
-
[16]
LaTeX formulas can be divided into inline and display; inline formulas use the ’$...$’ style, and display formulas use the ’$$...$$’ style
-
[18]
Do not insert any tables in the text
-
[19]
You can choose several of the following MarkDown syntax styles when creating: - Different levels of headings - Bold - Italic - Bold and italic - Underline - Superscript - Subscript - Lists - Unordered lists - Ordered lists
-
[20]
The style and organization of the entire text should be more varied, **do not always add a summary at the end**
-
[21]
The content of this text does not have to be complete; it can be a truncated content
-
[22]
Please only return the generated text content, do not return any other content, such as descriptions before and after. 16
-
[25]
Random seed is SEED Available text language and format styles: Exam paper, slides, academic paper, book, textbook, magazine, notes, newspaper, financial report Example: # The Future of Artificial Intelligence **Artificial Intelligence** (AI), as one of the most revolutionary technologies of the 21st century, is rapidly changing our way of life and work pa...
-
[26]
**Deep Learning**: Through multi-layer neural networks, AI can process complex data patterns. For example, Convolutional Neural Networks (CNN) perform excellently in image recognition, while Recurrent Neural Networks (RNN) have wide applications in Natural Language Processing (NLP)
-
[27]
A famous example is AlphaGo, which surpassed top human players by playing against itself
**Reinforcement Learning**: AI continuously optimizes the decision-making process through interaction with the environment. A famous example is AlphaGo, which surpassed top human players by playing against itself
-
[28]
## Application of Mathematical Formulas In AI research, mathematical formulas play a crucial role
**Transfer Learning**: AI can transfer knowledge from one task to another, improving the generalization ability of models. ## Application of Mathematical Formulas In AI research, mathematical formulas play a crucial role. For example, the training process of neural networks can be represented by the following formula: $$ \textLoss = \frac{1}{N} \sum_{i=1}...
-
[29]
Create the text based on the content of the table, around 300 words, and insert the table into the generated text
-
[30]
You can choose to use some of the following MarkDown syntax in the creation: - Different levels of headings - Bold - Italic - Bold Italic - Underline - Superscript 17 - Subscript - Lists - Unordered lists - Ordered lists
-
[31]
The style and organization of the whole text should be more varied, **do not always add a summary at the end**
-
[32]
The content of this text does not have to be complete, it can be a truncated content
-
[33]
Please return only the generated text, do not return any other content, such as descriptions before and after
-
[34]
Please return in English
-
[35]
Insert the table as it is, do not make any changes to the table, and there should be no line breaks or indentations between the html tags
-
[36]
The random seed is: SEED TABLE Prompt for Multi-column Please choose one of the topics given below and, based on the current random seed, write a MarkDown text with the following requirements:
-
[37]
Create content based on the theme "TOPIC", with a word count of around 600-800 words
-
[38]
The language and format style of the text should be chosen from the styles provided below
-
[39]
- Multi-line equation styles, such as equation&split, align, gather, alignat, etc
You may insert some Latex formulas, choosing from the following formula styles: - Matrix styles, such as matrix, array, pmatrix, bmatrix, vmatrix, Vmatrix, Bmatrix, cases, rcases, smallmatrix, subarray, etc. - Multi-line equation styles, such as equation&split, align, gather, alignat, etc. - Regular mathematical formula styles, such as frac, sum, etc
-
[40]
Latex formulas can be divided into inline and display; inline formulas use the ’$...$’ style, while display formulas use the ’$$...$$’ style
-
[41]
Both inline and display formulas can be inserted into the generated text
-
[42]
You may choose several MarkDown syntax options to use during creation: - Different levels of headings - Bold - Italic - Bold Italic - Underline - Superscript - Subscript - Lists - Unordered lists - Ordered lists
-
[43]
The style and organization of the entire text should be more varied; **do not always add concluding remarks at the end**
-
[44]
The content of this text does not have to be complete and can be a truncated piece
-
[45]
Please return only the generated text content, without any additional descriptions before or after
-
[46]
Do not indicate the chosen text language or format style in the returned text
-
[47]
Return the text in English
-
[48]
The random seed is: SEED
-
[49]
Divide this text into two paragraphs, with "x———-x" separating the paragraphs. Available text languages and format styles: Exam paper, slides, academic paper, book, textbook, magazine, notes, newspaper, financial report Here is an example: # The Wonderful World of Mathematics Mathematics, this ancient yet vibrant discipline, runs through the entire develo...
-
[2023]
In International conference on ma- chine learning, pages 19730–19742
Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In International conference on ma- chine learning, pages 19730–19742. PMLR. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report...
Pith/arXiv arXiv 2023
-
[2025]
arXiv preprint arXiv:2504.07491
Kimi-vl technical report. arXiv preprint arXiv:2504.07491. Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, and 1 others. 2024a. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan,...
Pith/arXiv arXiv 2024
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.