{"id":"712ff60d-f7c7-4e59-872c-1e351850cea4","arxiv_id":"2509.01215","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"POINTS-Reader uses a two-stage synthetic-data warm-up plus iterative self-improvement with rule-based filtering to train a 3B vision-language model for document conversion, outperforming larger models on OmniDocBench and Fox.","lead":"This paper trains a document conversion AI without using a large teacher model's outputs, by first generating synthetic documents and then having the model label real documents, filter its own errors, and retrain. It produces a compact 3B model that beats several larger open and commercial OCR models on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline SOTA comparison is not made under a common protocol: Table 5 mixes metrics from original papers, so the claimed margins over Mistral OCR/OLMOCR may be artifacts of evaluation differences.","rationale":"I read the paper in good faith: the two-stage pipeline is internally consistent, the ablations support the contribution of each component, and the model is publicly released. The central claim, however, is comparative: POINTS-Reader achieves state-of-the-art results and surpasses larger models. The most load-bearing evidence for that claim is Table 5, whose own footnote discloses that non-Qwen baselines were not run under the same evaluation settings. This is not a matter of consensus or taste; it is an internal evidentiary gap. The reader's weakest_assumption (PaddleOCR as a reliable text reference) is a plausible methodological weakness, but it is secondary to the evaluation inconsistency because even a PaddleOCR-biased training filter could still yield a model that outperforms baselines under a fair comparison. By contrast, if the baseline numbers are not comparable, the headline superiority claim is simply unverified. The concrete test—re-running all models under one protocol—would settle this directly. Since the reader's CONDITIONAL verdict already hinges on such re-verification, my analysis does not change the verdict; it sharpens the condition. I therefore recommend UNCHANGED, with the explicit condition that Table 5 be reproduced under a common evaluation harness.","tokens_in":18830,"tokens_out":5964,"duration_ms":69957,"concrete_test":"Run the official OmniDocBench evaluation harness on POINTS-Reader, OLMOCR-7B, GOT-OCR, Qwen2.5-VL-3B/7B, and, if API access permits, Mistral OCR, using identical input resolution, prompt template, max new tokens, and the same metric script and benchmark version. Compare overall OmniDocBench scores. If POINTS-Reader's 0.259 is no longer lower than Mistral OCR and OLMOCR under the common protocol, the 'surpasses' claim fails. For proprietary Mistral OCR, report results from at least three independent runs of the same harness and average them, or explicitly state the protocol deviation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim—that POINTS-Reader surpasses Mistral OCR, OLMOCR, and other larger models—rests on Table 5. However, the table's own note admits the baselines were not evaluated under a common protocol: 'The performance of the Qwen2.5-VL series is reported using the same evaluation settings as POINTS-Reader. For other methods, we use the metrics reported in their original papers, or, when unavailable, from subsequent works.' OmniDocBench scores are sensitive to the prompt template, output schema (POINTS-Reader emits unified HTML/LaTeX; OLMOCR and Mistral OCR emit native Markdown), max-token limits, image preprocessing, and the exact metric-normalization script. If the reported 0.268 for Mistral OCR or 0.326 for OLMOCR came from slightly different evaluation code, prompt design, or benchmark version, then the claimed margin over POINTS-Reader's 0.259 could shrink, vanish, or reverse. This is the load-bearing evidence for 'state-of-the-art'; without a common harness, the central comparative claim is not established. The PaddleOCR filtering issue is real but secondary: even if PaddleOCR biases the plain-text training signal, the model could still genuinely beat the baselines. The evaluation inconsistency directly undermines the comparison used to prove that it does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fully automated, distillation-free two-stage framework for document conversion. The Uniform Format Warm-up Stage (UWS) standardizes plain text, tables, and formulas into Markdown/HTML/LaTeX, uses an LLM to synthesize text and renders it through HTML templates, and fine-tunes POINTS-1.5. The Iterative Self-improvement Stage (ISS) runs the current model on DocMatix, filters outputs using PaddleOCR-based F1 for plain text, structural checks for tables, and LaTeX syntax checks for formulas, then retrains, repeating over iterations. The resulting 3B POINTS-Reader is reported to achieve 0.259 overall on OmniDocBench (en), outperforming Qwen2.5-VL-3B/7B, OLMOCR, and Mistral OCR, and approaching Qwen2.5-VL-72B. Ablations show monotonic gains from each data type and filter.","tokens_in":19090,"tokens_out":5619,"duration_ms":62072,"significance":"If the comparative numbers hold under a common protocol, the paper makes a meaningful contribution: it shows that a compact document-conversion model can be trained without teacher-model distillation, using synthetic warm-up plus iterative self-improvement with cheap rule-based filters. The paper ships a public model and provides consistent internal ablations, a scaling study, an analysis of the self-improvement loop, and an additional comparison against a Qwen2.5-VL-72B distillation baseline; these are strengths. The main barrier to accepting the headline claim is external: the state-of-the-art comparison in Table 5 mixes evaluation protocols, and the text filter's reference signal comes from a single traditional OCR engine, so the reported margins may be protocol artifacts or partly encode PaddleOCR's errors.","major_comments":[{"comment":"The central claim that POINTS-Reader surpasses Mistral OCR and OLMOCR is not established because the baselines were not evaluated under a common protocol. The table footnote states that only the Qwen2.5-VL series is re-run with the same settings; all other methods use metrics from their original papers or subsequent works. OmniDocBench scores are sensitive to the prompt template, output schema (POINTS-Reader emits unified HTML/LaTeX while OLMOCR/Mistral emit Markdown), token limits, and normalization scripts. The reported margins (0.259 vs. 0.326 OLMOCR, vs. 0.268 Mistral OCR) are small enough that protocol differences could reverse them. Please provide a common-harness rerun for the open baselines, or temper the SOTA claim.","section":"§3.3, Table 5"},{"comment":"The plain-text filter uses PaddleOCR as the reference for F1, so any systematic OCR error (formulas, page numbers, headers, footnotes, unusual fonts) is treated as an error in the model's output. In the ISS loop, this can select annotations that agree with PaddleOCR rather than with ground-truth page content. The paper's own F1-threshold ablation (Table 4) admits traditional OCR 'cannot recognize formulas' and that predictions omitting minor elements such as headers or page numbers may still be useful. This makes the filtering criterion misaligned with the stated goal of high-quality data. Please add a human or multi-OCR validation sample across iterations, and quantify how often the filter removes correct predictions or retains wrong ones.","section":"§2.2, Eqs. (1)–(3)"},{"comment":"The F1 threshold (0.90) and the aspect-ratio range [(2/5,5/2)] are selected by sweeping on the OmniDocBench test set. Since these hyperparameters are tuned on the benchmark used for the final headline numbers, the reported scores are an upper envelope. The risk is not that the ablations are uninformative, but that the specific choices may be overfit to OmniDocBench's English layout distribution. Please report the chosen thresholds from a held-out validation split, or show that the conclusions are stable across thresholds.","section":"§3.2.2, Tables 2 and 4"}],"minor_comments":[{"comment":"Inconsistent naming: 'DocMatrix' appears in §2.2 and §3.2.2 while DocMatix is used elsewhere. Also typos: 'Instuct' (model training), 'PONTS-Reader' (Appendix A.1), 'KaTex' reference formatting.","section":"Throughout"},{"comment":"The text says 'we evaluate all models on four benchmarks,' but Table 5 lists only OmniDocBench and Fox. Clarify whether 'four benchmarks' refers to four evaluation suites or to the sub-metrics within OmniDocBench.","section":"§3.3"},{"comment":"Caption (c) repeats 'the number of retained samples containing tables increases after filtering' from (b); presumably one should say 'formulas'.","section":"Figure 6"},{"comment":"Figure references appear swapped: the paragraph on loading weights refers to Figure 8, but the relevant figure is numbered 7; the later paragraph on UWS data inclusion refers to the figure below, numbered 8.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"I would make the common-protocol comparison a requirement rather than a suggestion. The paper is otherwise a solid data-centric contribution; if the authors can rerun open baselines (OLMOCR, GOT-OCR, Qwen2.5-VL family) in the same harness and report the exact prompts/eval code, I would be willing to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The two-stage pipeline here is genuinely worth knowing about: synthetic warm-up with a unified output format, then iterative self-improvement on real documents with rule-based filtering. The components aren't individually new, but the specific combination and the public 3B model are a real contribution. The ablations are internally consistent and show each piece doing something. I'd believe the method improves document conversion over the stable baseline.\n\nThe soft spots are real, though. The biggest one is Table 5. The note admits that for most baselines—Mistral OCR, OLMOCR, GOT-OCR—they're pulling numbers from original papers or subsequent works, while only the Qwen2.5-VL series is re-run under their own settings. OmniDocBench is sensitive to prompt template, output schema, token limits, and normalization. So the claimed margins over Mistral OCR (0.268 vs 0.259) and OLMOCR (0.326 vs 0.259) could shrink, vanish, or reverse in a common harness. That's the load-bearing evidence for \"surpassing larger models.\"\n\nSecond, the F1 filter uses PaddleOCR as ground truth for plain text. That biases training toward PaddleOCR's errors, especially on headers, formulas, and unusual fonts. The paper shows improvement with filtering, but part of that improvement likely reflects convergence to PaddleOCR's output rather than pure reading ability. It doesn't kill the method, but it tempers the claim that this is fully \"distillation-free\" in spirit.\n\nThird, the F1 threshold and aspect-ratio range are tuned by ablating on OmniDocBench directly. That's a form of eval-set overfitting; the reported gains from those choices are probably inflated.\n\nNone of this means the central approach is broken. The iterative self-improvement plus rule-based filtering is sensible, and the authors are transparent about many limitations (English-only, printed fonts). The model is public. But the headline SOTA comparison needs a re-run under a single evaluation protocol before I'd trust the specific margins.\n\nThis deserves a serious referee. The editor should ask for a common evaluation harness for all baselines, and ideally a held-out validation set for any hyperparameter choices. I'd engage with the work; I just wouldn't take the leaderboard at face value.","headline":"The self-improvement pipeline is real and the ablations are solid, but the SOTA claim rests on a comparison table that mixes evaluation protocols, so the headline margins over Mistral OCR and OLMOCR are not yet established.","tokens_in":19621,"tokens_out":2050,"would_cite":true,"duration_ms":25583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully automated, distillation-free two-stage pipeline trains a 3B vision-language model for document conversion that outperforms larger public and proprietary models.","keywords":["document conversion","vision-language model","distillation-free training","synthetic data generation","self-improvement","rule-based filtering","OCR","table and formula extraction"],"falsifier":"Take pages where PaddleOCR is known to be wrong—formulas, footnotes, headers, unusual fonts—and compare POINTS-Reader's output to human transcription. If the model reproduces PaddleOCR's errors more than a version trained without the F1 filter does, then the measured improvement is partly alignment to the reference rather than better document reading. A simpler check: score POINTS-Reader against OmniDocBench's human-based ground truth and see if the text-metric gap over Qwen2.5-VL-7B persists.","tokens_in":18696,"feed_emoji":"📄","tokens_out":6684,"duration_ms":71322,"temperature":0.7,"pith_summary":"Document conversion means turning a page image into structured text: plain paragraphs, tables, and mathematical formulas. The paper claims that this can be done well without distilling from a stronger teacher model. Its pipeline first generates a large set of synthetic pages by asking a language model to write text and rendering it into images with HTML templates, using one standardized output format per element: Markdown for text, HTML for tables, LaTeX for formulas. A 3B vision-language model is fine-tuned on those pairs, then used to annotate real documents. A set of rule-based filters keeps only outputs that match a traditional OCR engine's plain-text reading, have structurally consistent tables, and contain syntactically valid formulas; retraining on the filtered pseudo-annotations is repeated several times. The final model is reported to reach a 0.259 overall score on OmniDocBench, better than Qwen2.5-VL-3B, Qwen2.5-VL-7B, OLMOCR, and Mistral OCR, and close to the much larger Qwen2.5-VL-72B.","feed_headline":"A 3B model with no teacher beats larger document OCR models","feed_subtitle":"Synthetic-data warm-up plus iterative rule-filtered self-training reaches 0.259 on OmniDocBench.","key_machinery":"The engine of the method is the iterative self-improvement stage, powered by three rule-based filters. Plain text is filtered by an F1 score against PaddleOCR, tables are checked for consistency of cell counts across rows and columns, and mathematical formulas are checked only for LaTeX syntactic validity. These filters deliberately do not judge semantic content: they reject hallucinations, broken table structures, and malformed formulas without needing a learned teacher. A warm-up stage on synthetic pages rendered from HTML templates gives the model a strong initial grasp of the unified output formats, and each self-improvement iteration re-initializes from the pretrained backbone rather th","core_discovery":"The paper's central claim is that a closed loop of self-annotation and cheap deterministic filtering can adapt a document-conversion model to real-world data without any teacher model. The key evidence is the final 3B model, POINTS-Reader: 0.259 overall on OmniDocBench (lower is better) against 0.390 for Qwen2.5-VL-3B, 0.331 for Qwen2.5-VL-7B, 0.326 for OLMOCR, and 0.268 for Mistral OCR. On the Fox English benchmark it reaches 0.023 normalized edit distance, ahead of all listed end-to-end models. The authors also report that the model outperforms a student trained by direct distillation from Qwen2.5-VL-72B, showing that the bottleneck in this setting is data quality rather than model capacit","pith_inferences":["Because the plain-text filter uses PaddleOCR as the reference, the model is effectively trained to agree with PaddleOCR; if PaddleOCR has systematic errors on headers, footnotes, formulas, or unusual fonts, those errors are likely baked into the training signal. A human-verified test set would show how much of the gain is genuine reading versus alignment to this reference.","The table and formula filters validate only structure and syntax, so the model may produce tables and formulas that look right but are semantically wrong; the reported improvements on those metrics may partly reflect learning formatting conventions, not content understanding.","The same recipe—synthetic warm-up plus self-generated data filtered by deterministic checks—could transfer to other structured extraction tasks, such as charts, forms, or slides, where semantic annotation is hard but structural validity is easy to check.","Since the final data is heavily plain-text (about 90 percent), further gains may depend on diversifying rare layouts and elements rather than scaling volume or rebalancing sample types."],"forward_implications":["Teacher-free training removes the ceiling imposed by a teacher's own errors: a student no longer inherits the teacher's hallucinations, missed tables, or wrong table structures.","A 3B model reaches near-72B accuracy on OmniDocBench overall and beats 72B on tables, so compact end-to-end converters are viable for deployment.","Because the filters ignore semantic correctness of tables and formulas, even cheap structural checks (cell-count consistency, LaTeX parse) can drive substantial gains.","The final iteration yields about 1.1M real-world image-text pairs with HTML tables and no distillation, a reusable resource for further training.","Synthetic data alone plateaus around 800k samples; the real-world self-improvement loop is what pushes performance further."],"supporting_citations":[{"why":"supplies the POINTS-1.5 base model and the two-stage training recipe used throughout.","marker":"(Liu et al., 2024e)"},{"why":"defines the Qwen2.5-VL baselines, including the 72B teacher used in the direct-distillation comparison.","marker":"(Bai et al., 2025)"},{"why":"PaddleOCR provides the reference text for the F1-score plain-text filter in the self-improvement stage.","marker":"(Du et al., 2020)"},{"why":"OmniDocBench supplies the overall benchmark score that drives ablations and the main comparison table.","marker":"(Ouyang et al., 2024)"},{"why":"DocMatix contributes the two-million-page real-world corpus on which self-annotations are generated and filtered.","marker":"(Laurençon et al., 2024)"},{"why":"PubTabNet supplies complex real tables that are inserted into synthetic text to increase structural diversity.","marker":"(Zhong et al., 2020)"},{"why":"provides the F1-score filtering method for plain text that the paper adapts.","marker":"(Yang et al., 2024b)"},{"why":"cited as the inspiration for the rule-based filtering strategies used in the self-improvement stage.","marker":"(Guo et al., 2025)"},{"why":"OLMOCR is both a 7B baseline and an example of a distillation-built dataset that the paper contrasts with its own.","marker":"(Poznanski et al., 2025)"}],"fun_headline_variants":["Self-training beats teacher distillation for document AI","No teacher needed: 3B model tops document OCR","Iterative self-annotation outdoes larger models in doc parsing","Synthetic warm-up plus self-correction wins document conversion","POINTS-Reader: zero distillation, top scores on OmniDocBench"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole pipeline assumes PaddleOCR's output is a trustworthy reference for what the page actually says; if PaddleOCR systematically misreads certain elements, those misreadings become the training target.","fun_headline_variants_meta":{"raw":{"variants":["Self-training beats teacher distillation for document AI","No teacher needed: 3B model tops document OCR","Iterative self-annotation outdoes larger models in doc parsing","Synthetic warm-up plus self-correction wins document conversion","POINTS-Reader: zero distillation, top scores on OmniDocBench"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1149,"prompt_tokens":837,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":581,"tokens_out":312,"duration_ms":4388,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:44:14.246414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take pages where PaddleOCR is known to be wrong—formulas, footnotes, headers, unusual fonts—and compare POINTS-Reader's output to human transcription. If the model reproduces PaddleOCR's errors more than a version trained without the F1 filter does, then the measured improvement is partly alignment to the reference rather than better document reading. A simpler check: score POINTS-Reader against OmniDocBench's human-based ground truth and see if the text-metric gap over Qwen2.5-VL-7B persists.","supporting_citations":[],"review_version":1}