{"id":"8e8be7a5-b669-4e40-84f5-4682b38d1081","arxiv_id":"2412.12902","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DoPTA pre-trains a vision encoder by aligning image patches with OCR text embeddings using IoU-guided soft labels, improving document layout analysis without OCR at inference.","lead":"The paper introduces DoPTA, a document image encoder pre-trained with a patch-text alignment loss that uses OCR word boxes as soft supervision. It reports improvements on document layout analysis benchmarks while staying OCR-free at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLIP initialization confounds the comparison to from-scratch baselines, so the claimed efficiency and outperformance advantages are not cleanly attributable to the proposed alignment loss.","rationale":"The reader's weakest_assumption—that CLIP initialization conflates the benefit of the proposed loss with the benefit of starting from powerful pre-trained weights—is exactly the most load-bearing concern. The central claims are (a) consistent outperformance over larger models and (b) significantly lesser pre-training compute. Both rest on comparisons to DiT and VGT, which are trained from scratch. Because DoPTA starts from CLIP, any comparison that does not control for initialization cannot separate the method's contribution from CLIP's. The ablation Table 5 partially addresses this by including a CLIP baseline, but it is at 224 resolution and only 160k steps; the headline numbers are at 512 and 250k steps, so the confound is not resolved. The paper deserves credit for a clear ablative design showing that the alignment loss does improve over its own CLIP baseline, and the qualitative heatmaps in the appendix provide support for the patch-text alignment working as intended. However, the efficiency claim is overstated because the compute to pre-train CLIP is externalized, and the downstream SOTA claims are not supported by statistical significance (e.g., FUNSD has only 50 test images). These issues do not invalidate the method, but they do mean the paper should be accepted conditionally, with the requested CLIP-only control experiment (or code release) as the condition. The reader's verdict already reflects this, so the verdict should remain CONDITIONAL (unchanged).","tokens_in":16107,"tokens_out":6837,"duration_ms":60213,"concrete_test":"Fine-tune the off-the-shelf CLIP ViT-B/16 (without any DoPTA pre-training) on D4LA at 512x512 using the same Cascade R-CNN pipeline, 60k fine-tuning steps, and hyperparameters as reported in Sec. 4.3.2, and compare its mAP to DoPTA's 70.72. If the CLIP-only model is within roughly 1 mAP of DoPTA, then the alignment loss adds little beyond initialization and the claimed superiority over from-scratch baselines is mostly an initialization effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DoPTA \"consistently outperforms larger models, while using significantly lesser pre-training compute\" is supported by comparisons to DiT and VGT, which are trained from scratch. However, Sec. 4.1 states that both the DoPTA encoder and the text encoder are initialized from CLIP-ViT-B/16 weights. The ablation in Table 5 shows that a fine-tuned CLIP baseline (no DoPTA pre-training) already achieves 93.3 mAP on PubLayNet and 64.5 mAP on D4LA at 224 resolution, versus 94.62 and 67.92 for DoPTA at the same resolution and 160k pre-training steps. Since the main results are reported at 512 resolution, and CLIP features generally improve with resolution, the reported 94.9/70.72 on PubLayNet/D4LA may be substantially inherited from CLIP rather than produced by the patch-text alignment loss. The \"significantly lesser pre-training compute\" claim also excludes the compute used to train CLIP on 400M image-text pairs, making the comparison inequitable. The paper never trains DoPTA from scratch nor initializes DiT or VGT with CLIP weights, so the isolated contribution of the alignment loss is not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DoPTA, a ViT-B/16 document image encoder pre-trained on IIT-CDIP with two objectives: a fine-grained TextToPatch contrastive loss in which each OCR token's distribution over image patches is supervised by the IoU between the token bounding box and each patch, and a masked autoencoding reconstruction loss. The text encoder and image decoder are used only during pre-training, so downstream tasks use only the image encoder. The authors evaluate on RVL-CDIP, PubLayNet, D4LA, M6Doc, and FUNSD, reporting state-of-the-art results on D4LA, M6Doc, and FUNSD and on-par PubLayNet performance with DiT-L, while using about 250k pre-training steps.","tokens_in":16381,"tokens_out":4767,"duration_ms":43388,"significance":"If the reported gains are attributable to the proposed objective, DoPTA is a useful OCR-free document encoder: it improves over the CLIP-initialized baseline in the 224/160k ablation of Table 5, outperforms DiT-B and DiT-L on several benchmarks with fewer parameters, and avoids OCR latency at inference. The paper's strengths are the simple and clearly specified loss, the controlled ablation in Table 5 showing that each loss component adds value, and the use of public benchmarks with the same detection and fine-tuning pipeline as DiT/VGT. The main caveat is attribution: because both the image and text encoders are initialized from CLIP-ViT-B/16 while DiT and VGT are trained from scratch, the claimed efficiency advantage over those baselines is not cleanly separated from CLIP initialization.","major_comments":[{"comment":"The claim that DoPTA 'consistently outperforms larger models, while using significantly lesser pre-training compute' is confounded by the CLIP initialization. Section 4.1 states that both the DoPTA encoder and the text encoder are initialized from CLIP-ViT-B/16 weights, whereas DiT and VGT are trained from scratch. Table 5 shows that the CLIP baseline alone already achieves 93.3 mAP on PubLayNet and 64.5 mAP on D4LA at 224 resolution and 160k steps, against 94.62 and 67.92 for the full DoPTA at the same setting. Since the headline results are reported at 512 resolution and 250k steps, and no CLIP baseline is reported at that setting, the 94.9/70.72 numbers on PubLayNet/D4LA may be substantially inherited from the pretrained CLIP weights. To support the efficiency and outperformance claims, the authors should report a CLIP-initialized baseline at 512 resolution with the same detection setup, and ideally also train DoPTA from scratch or initialize DiT/VGT with CLIP weights, so that the isolated contribution of the patch-text alignment loss is demonstrated. The comparison of pre-training steps alone also excludes the compute used to train CLIP on hundreds of millions of image-text pairs.","section":"§4.1, Table 5"},{"comment":"The headline statement 'consistently outperforms larger models' is contradicted by the PubLayNet results in Table 2, where DoPTA achieves 94.9 mAP and DiT-L also achieves 94.9 mAP, and the text in §4.3.1 explicitly calls this performance 'on-par with DiT-L.' Since 'consistently outperforms' appears in the abstract and in the introduction, it should be qualified to a claim of competitive or improved performance, with the exact benchmark-by-benchmark comparison stated clearly.","section":"Abstract and §4.3.1, Table 2"}],"minor_comments":[{"comment":"The baseline 'UDoc []' in Table 2 has an empty citation and no parameter count; please add the missing reference and clarify the experimental setup for that row.","section":"Table 2"},{"comment":"The notation bbox(I_j) in Eq. (2) is never defined for image patches; please state explicitly that it denotes the pixel region covered by patch j and explain how partial overlaps between text bounding boxes and patch regions are computed.","section":"§3.2, Eq. (2)"},{"comment":"The row labels in Table 5 are easy to misread; since all rows use CLIP initialization, consider renaming the rows to 'CLIP init only', 'Reconstruction only', 'Patch-text alignment only', and 'Both losses', and state in the caption that all rows share the same CLIP initialization.","section":"Table 5"},{"comment":"The inference-time comparison adds EasyOCR parsing time to VGT's FPS but not to DoPTA's; please state explicitly in the text that OCR time is included only for OCR-based methods and note that the reported speedup depends on the specific OCR engine and hardware.","section":"§4.5"},{"comment":"The statement that the Equation category has 'only 2/3 samples in total' is ambiguous and appears inconsistent with the category-wise mAP reported in Table 3; please clarify whether this means two or three ground-truth instances, and if so, consider reporting the confidence intervals or variance for that category.","section":"Appendix C"},{"comment":"There are several formatting and typographical issues, including inconsistent spacing in the model name ('DOPTA' vs 'D OPTA') and the misspelling 'signifcantly' in Section 5; please proofread the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is promising, but the CLIP-initialization confound is the main risk. If the authors can provide a CLIP baseline at the same resolution as the main results or a from-scratch training comparison, the contribution would be much more convincing; otherwise the efficiency claim should be substantially weakened. The abstract also needs to be aligned with the actual PubLayNet result, where DoPTA ties DiT-L rather than outperforming it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on DoPTA. The new piece is the IoU-guided patch-text alignment loss: each OCR token's embedding is pulled toward image patches it overlaps, with the ground-truth soft label being the fraction of the token box covered by each patch. That's a clean adaptation of FILIP/GLIP-style token-level contrastive learning to documents, and it's a sensible way to inject text semantics into a vision encoder without needing OCR at inference. The paper is also honest about its lineage and includes a useful ablation (Table 5) showing that both the alignment loss and the MAE-style reconstruction improve on the raw CLIP checkpoint: on D4LA, 67.92 vs. 64.5 mAP at 224 resolution, 160k steps. The FPS analysis (OCR-free inference gives roughly 12x the throughput of VGT) is practically relevant. Good that they ran a PixParse pre-training variant in the appendix, even if CDIP wins.\n\nThe soft spots are real, though. Both encoders start from CLIP-ViT-B/16 weights, while DiT and VGT are trained from scratch. So the 'significantly lesser pre-training compute' claim silently excludes the cost of training CLIP on 400M image-text pairs. The ablation makes the confound concrete: a fine-tuned CLIP baseline already gets 93.3 on PubLayNet, and DoPTA gets 94.62 at the same resolution and 160k steps. The new loss adds value, but the headline numbers against DiT/VGT are partly inherited from CLIP. To isolate the contribution, they'd need a from-scratch run or CLIP-initialized baselines. Also, 'consistently outperforms larger models' is too strong: on PubLayNet, DoPTA ties DiT-L at 94.9. No error bars, no code release, and no ablation at the final 512/250k setting. The M6Doc SOTA could be checked against the original benchmark, but their numbers are plausible.\n\nOverall, the central idea is not broken. It's a modest but useful contribution to document layout analysis, and the ablations are the right kind of evidence. The paper needs a fairer comparison, better wording, and open code before I'd trust the efficiency claim. It deserves a serious referee, and I'd send it out with a request for revision rather than desk-reject it.","headline":"A sensible IoU-guided patch-text alignment loss for document pre-training, with honest ablations; the CLIP-initialization confound and a PubLayNet tie blunt the efficiency and outperformance claims, but the contribution is worth refereeing.","tokens_in":16852,"tokens_out":4316,"would_cite":true,"duration_ms":38514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By aligning image patches with OCR text during pre-training, a compact 85M-parameter document encoder outperforms larger OCR-dependent models on layout analysis, text detection, and classification while needing no OCR at inference.","keywords":["document layout analysis","patch-text alignment","contrastive pretraining","OCR-free document understanding","vision transformer","masked image reconstruction","IoU-guided alignment","document AI"],"falsifier":"Train DoPTA from random initialization for the same 250k steps on IIT-CDIP and evaluate on D4LA and FUNSD; if its score drops to the DiT level or below, or if CLIP-initialized DiT matches DoPTA, then initialization rather than patch-text alignment is carrying the result.","tokens_in":15928,"feed_emoji":"📄","tokens_out":6735,"duration_ms":57326,"temperature":0.7,"pith_summary":"The paper claims that document layout understanding improves when a vision encoder is pre-trained to align its image patches with the words written on the page, even though no OCR is needed when the model is later used. It introduces DoPTA, an 85M-parameter ViT-style encoder trained on IIT-CDIP with two losses: an IoU-guided patch-text contrastive loss that connects each text token to the image patches its bounding box covers, and a masked reconstruction loss for non-textual visual content. With only 250k pre-training steps, DoPTA reports state-of-the-art results on D4LA (70.72 mAP) and FUNSD (94.73 F1), matches a much larger DiT-L on PubLayNet, and runs about 12 times faster than OCR-based VGT at inference because OCR is not invoked. The significance is that textual semantics can be injected into a purely visual encoder at pre-training time, avoiding OCR latency and OCR error propagation downstream.","feed_headline":"Document encoder DoPTA beats bigger layout models with less training","feed_subtitle":"SOTA layout scores plus ~12x faster inference come from aligning image patches with OCR text during pre-training.","key_machinery":"The load-bearing object is the TextToPatch contrastive loss. For each text token embedding $X_i^T$ and each image patch embedding $X_j^I$, the loss is an asymmetric cross-entropy\n$$L_i = -\\sum_{j=1}^N Y(T_i,I_j)\\,\\log \\frac{\\exp(\\$\\lambda$ s_{i,j})}{\\sum_k \\exp(\\$\\lambda$ s_{i,k})},$$\nwith similarity $s_{i,j}=X_i^T\\cdot X_j^I$ and ground-truth weight $Y(T_i,I_j)=|\\mathrm{bbox}(I_j)\\cap \\mathrm{bbox}(T_i)|/|\\mathrm{bbox}(T_i)|$, the fraction of the text token's bounding box covered by that patch. It is averaged over all text tokens and combined with a MAE-style masked reconstruction loss. This loss converts OCR text geometry into a per-patch learning signal, and only the image encoder is kept for downstream tasks.","core_discovery":"On its own terms, the paper's discovery is that a fine-grained, spatially grounded contrastive alignment between text tokens and image patches is a strong pre-training signal for document images. The ground truth for alignment is not a caption but the geometry of the text: for each OCR token, the target distribution over image patches is the fraction of that token's bounding box falling in each patch, and the model is trained to make its patch similarities match that distribution. Adding a masked-pixel reconstruction head on top further helps regions such as figures and diagrams that carry no text. The trained image encoder alone, without OCR at inference, reaches 94.12% on RVL-CDIP, 94.9 mAP on PubLayNet, 70.72 mAP on D4LA, 69.5 mAP on M6Doc, and 94.73 F1 on FUNSD, outperforming the much larger DiT-L and the OCR-dependent VGT on several benchmarks while using fewer parameters and a shorter pre-training schedule.","pith_inferences":["The headline comparison to DiT and VGT conflates the alignment loss with CLIP initialization; the paper's own CLIP baseline shows the loss adds value over CLIP, but a from-scratch DoPTA is needed to know how much of the SOTA gap over from-scratch baselines comes from the loss versus the initialization.","The reported pre-training budget excludes the cost of training the CLIP weights both encoders start from, so including that cost would shrink the claimed compute advantage.","If the gains survive training from random initialization, the same IoU-weighted soft alignment could be ported to other dense visual tasks where weak spatial text labels exist, such as scene-text detection or chart and figure parsing.","A natural next experiment is to generate text descriptions of non-textual regions so that the alignment loss also covers figures and tables, not only OCR words."],"forward_implications":["Document layout detectors can get the semantic benefit of reading text without paying OCR cost at inference, since the text is consumed once during pre-training.","Smaller encoders pre-trained this way can match or beat much larger document transformers, shifting the driver of document AI performance from model size to pre-training objective.","The roughly 12x inference speedup over OCR-based VGT on D4LA makes high-throughput document parsing more practical.","Classes whose identity is carried by their wording (titles, questions, dates, authors) improve most, consistent with the loss aligning patches to words.","Combining the alignment loss with masked reconstruction extends gains to non-textual layout elements like figures and tables."],"supporting_citations":[{"why":"Supplies the CLIP-ViT-B/16 initialization and the contrastive image-text training paradigm that the patch-text alignment loss extends.","marker":"[36]"},{"why":"Supplies the fine-grained token-patch alignment idea that the IoU-guided TextToPatch loss adapts to document images.","marker":"[50]"},{"why":"Supplies the masked reconstruction objective and shallow decoder used for the auxiliary reconstruction loss.","marker":"[17]"},{"why":"DiT is the main OCR-free baseline and evaluation setting for RVL-CDIP, PubLayNet, and FUNSD that DoPTA compares against.","marker":"[25]"},{"why":"VGT is the OCR-based state-of-the-art baseline and introduced the D4LA benchmark on which DoPTA reports new state-of-the-art results.","marker":"[12]"},{"why":"Donut provides the OCR-free encoder-decoder baseline and the encoder-only comparison used in classification.","marker":"[24]"},{"why":"EasyOCR provides the word-level text and bounding boxes used to generate the patch-text alignment targets during pre-training.","marker":"[21]"},{"why":"IIT-CDIP is the pre-training corpus, selected for its scale and for parity with the baselines.","marker":"[39]"}],"fun_headline_variants":["Patch-text alignment trims OCR-free document layout training","DoPTA: OCR-free layout model tops benchmarks with less compute","Aligning text patches boosts document layout AI without OCR","Smaller DoPTA model outdoes giants, skips OCR at runtime"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the improvement is caused by the new alignment loss, not by the fact that DoPTA starts from pre-trained CLIP weights while the main baselines start from scratch; the reported training budget also leaves out the cost of training those CLIP weights.","fun_headline_variants_meta":{"raw":{"variants":["Patch-text alignment trims OCR-free document layout training","DoPTA: OCR-free layout model tops benchmarks with less compute","Aligning text patches boosts document layout AI without OCR","Smaller DoPTA model outdoes giants, skips OCR at runtime"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3053,"prompt_tokens":918,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2064}},"tokens_in":534,"tokens_out":2135,"duration_ms":12501,"temperature":1.0,"reasoning_tokens":2064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:36:32.975517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DoPTA from random initialization for the same 250k steps on IIT-CDIP and evaluate on D4LA and FUNSD; if its score drops to the DiT level or below, or if CLIP-initialized DiT matches DoPTA, then initialization rather than patch-text alignment is carrying the result.","supporting_citations":[{"cited_title":"FILIP: Fine-grained interactive language- image pre-training","cited_arxiv_id":null,"evidence_quote":"Supplies the fine-grained token-patch alignment idea that the IoU-guided TextToPatch loss adapts to document images."},{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Supplies the masked reconstruction objective and shallow decoder used for the auxiliary reconstruction loss."},{"cited_title":"Dit: Self-supervised pre-training for docu- ment image transformer","cited_arxiv_id":null,"evidence_quote":"DiT is the main OCR-free baseline and evaluation setting for RVL-CDIP, PubLayNet, and FUNSD that DoPTA compares against."},{"cited_title":"Vision grid transformer for document layout analysis","cited_arxiv_id":null,"evidence_quote":"VGT is the OCR-based state-of-the-art baseline and introduced the D4LA benchmark on which DoPTA reports new state-of-the-art results."},{"cited_title":"Ocr-free document understanding transformer","cited_arxiv_id":null,"evidence_quote":"Donut provides the OCR-free encoder-decoder baseline and the encoder-only comparison used in classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EasyOCR provides the word-level text and bounding boxes used to generate the patch-text alignment targets during pre-training."},{"cited_title":"Complex document information processing (cdip) dataset, 2022","cited_arxiv_id":null,"evidence_quote":"IIT-CDIP is the pre-training corpus, selected for its scale and for parity with the baselines."}],"review_version":1}