REVIEW 3 major objections 6 minor 30 references
Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that OCR-based visual document understanding models are far less robust than clean benchmarks suggest: budgeted adversarial shifts to bounding boxes, pixels, and text can cut F1 or ANLS by as much as 29.18%.
desk verdict Useful benchmark idea, but the flagship PGD layout attack is never shown to be realizable as a document; worth reviewing but needs major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a learned bounding-box predictor $g_\theta$ that maps token embeddings to box parameters (center coordinates plus log width and log height), trained with SmoothL1 and GIoU losses. This predictor makes layout embeddings differentiable, allowing Projected Gradient Descent on the token embeddings under an mIoU-budget loss with projection onto the feasible set $\text{IoU} \ge \tau$. Text and pixel modules bound character edits and use RoDLA-style document transforms (blur, noise, occlusion) to keep perturbations visually coherent.
What would settle it
Take one of the PGD line-level adversarial embeddings from FUNSD, render or search for a physical document image whose OCR output produces that embedding, feed the image through the full OCR-to-LayoutLMv3 pipeline, and measure the F1 drop; if the drop largely disappears, the vulnerability claim does not survive the image domain.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a budgeted, multi-modal attack can reveal severe vulnerabilities in OCR-based VDU models: up to a 29.18% F1 drop on FUNSD under the compound BBox + Pixel + Text scenario. Line-level attacks consistently outperform word-level attacks, PGD-based layout perturbations outperform random shift baselines even under tight IoU budgets, Unicode diacritic text modifications cause larger degradation than random character replacement, and PGD-generated examples transfer across model architectures, including to text+layout models that have no image input.
Load-bearing premise
The attack is optimized in the model's token-embedding space using a trained box predictor, but the paper never constructs or verifies a real document image whose OCR output would induce that exact embedding, so the reported F1 drops could be an artifact of the embedding space rather than a realizable document perturbation.
Editorial extensions
If this is right
- Production OCR-based document systems should treat clean F1 scores as an upper bound, since plausibility-bounded layout, pixel, and text edits can induce double-digit F1 drops.
- Because line-level attacks are consistently stronger than word-level attacks, OCR engines that merge word boxes into lines expand the attack surface and should be hardened first.
- PGD-generated layout perturbations transfer to architectures with no visual input, meaning a single crafted document can hurt multiple deployed models without per-model optimization.
- Stricter IoU budgets reduce but do not eliminate attack strength: on FUNSD, PGD still produced a 6.5% F1 drop at IoU 0.9 while random shifts fell to 0.54%.
- Unicode diacritic text edits are more damaging than random character replacement, so text-based defenses must target visually confusable glyphs, not just typo-level noise.
Reading between the lines
- The attack optimizes embeddings in model token space via the box predictor but never renders a real document image whose OCR would produce those embeddings; whether the reported drops survive a full image-to-OCR-to-model pipeline is an open question the paper does not answer.
- The IoU budget measures geometric overlap, not semantic plausibility, so an even stronger real-world attack might exist that keeps boxes highly overlapping while moving them to layout positions that are still plausible for the document type.
- If line-level attacks are this effective, OCR systems that pre-aggregate word boxes into lines are converting a robustness problem into a general layout-embedding problem; the paper's results imply that defense should happen at the line grouping stage, not only at the classifier.
- The transfer results to text+layout models suggest the vulnerability lives largely in the layout embedding itself rather than the visual features, which points to alignment or preprocessing defenses on box tokens as the most direct next step.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified multi-modal adversarial attack framework for OCR-based Visual Document Understanding (VDU) models, combining bounding-box, pixel, and text perturbations under explicit budgets. The method trains a differentiable bounding-box predictor to enable PGD attacks on token embeddings, defines six attack scenarios at word and line granularity, and reports F1/ANLS drops across FUNSD, CORD, SROIE, and DocVQA on LayoutLMv2, LayoutLMv3, ERNIE, GeoLayoutLM, LayTextLLM, LLaMA, and GPT-4.1 mini. The central claims are that line-level and compound (BBox+Pixel+Text) attacks cause the largest degradation, that PGD outperforms random shift, and that Unicode text attacks are more damaging than random edits.
Significance. If the attack framework were fully realized and the results complete, this would be a valuable robustness benchmark for OCR-based VDU systems, which are widely deployed in document-processing pipelines. The paper's strengths include the breadth of scenarios (six), the coverage of multiple architectures and datasets, the inclusion of a learned differentiable bbox predictor for gradient-based layout attacks, and a transparent limitations section that acknowledges white-box scope and the exclusion of OCR-free models. The ablation on IoU budgets and the comparison of Unicode versus random text edits are informative. However, the central attack is never demonstrated to correspond to a realizable document perturbation, and the key results table is incomplete, both of which currently block acceptance.
major comments (3)
- [§3.3, §3.6, Tables 3–7] The attack is optimized in continuous token-embedding space, and the layout budget is enforced only on boxes predicted by g_theta(e~). No step in the pipeline constructs a real document image whose OCR output would reproduce e~, nor a discrete OCR string and concrete bounding-box edit that realizes the perturbation. The pixel and text modules in §3.4 are applied after the fact, but the layout channel is never realized as an actual document. Consequently, the reported F1/ANLS drops measure sensitivity of the frozen model to internal embedding perturbations, not robustness of OCR-based VDU systems to the claimed budgeted, realistic input attacks. This is the load-bearing missing support for the abstract's central claim of 'up to 29.18% F1 drop' under a plausible perturbation. Please either render the perturbed boxes onto the document and re-run OCR, or otherwise demonstrate that the embedding-space perturbation can be realized as a document whose OCR output induces e~.
- [Table 3] The key results table contains the note 'PGD values will be filled in separately if available,' and the DocVQA columns lack PGD results entirely. Yet §4.3.1 claims 'PGD-based attacks consistently yield greater performance degradation than Random Shift' across all investigated datasets, and §4.3.3 uses these numbers for the KIE-vs-VQA task comparison. Without complete PGD entries, the headline comparison is not supported as presented. Please complete the table or explicitly report which PGD values could not be obtained and why.
- [§4.2] The BBox predictor g_theta is trained on the same four datasets used for evaluation, but the paper never states the train/test split between documents used to fit g_theta and documents used to generate and evaluate attacks. If the predictor is trained on the test split, the mIoU values in Table 2 and the resulting attack effectiveness are inflated, and the reported transferability results are not trustworthy. Please specify the split and, if necessary, retrain the predictor on a separate held-out set before generating attacks.
minor comments (6)
- [Table 1 caption] The caption says the table shows 'the number of document images and bounding boxes,' but Table 1 lists only bounding-box counts; no document-image counts are given.
- [§3.3, Eq. (1)] The objective L_adv = L_task - λ_box (1 - IoU(hat B, tilde B)) uses the predicted clean boxes hat B rather than the original ground-truth boxes B. As written, the budget is relative to the predictor's output, not to the actual clean boxes, which is inconsistent with the threat-model definition in §3.1.
- [§3.3] The text says the PGD update 'projects the resulting boxes back into the feasible set {IoU≥τ}', but no projection operator is specified for embedding space. It is unclear how one projects a token embedding so that the predicted boxes satisfy the IoU constraint.
- [Table 2 and Table 5] GeoLayoutLM rows in Table 2 are all dashes, yet Table 5 reports GeoLayoutLM PGD transfer results. Please clarify whether these are transfer attacks generated with the LayoutLMv3 predictor and whether the asterisk '*Word level' applies to all GeoLayoutLM entries.
- [§4.3] All results are averaged over 5 random seeds, but no standard deviations or confidence intervals are reported. Given that some reported differences are small (e.g., Table 7, PGD at IoU 0.75 vs 0.9: 6.60 vs 6.50), the claimed 'consistently' and 'more effective' trends need variance estimates to be statistically assessed.
- [References] The Unicode diacritic attack is attributed to (Boucher et al., 2023), but the reference entry lacks a year and venue, and the genetic-optimization details are not described in §3.4.
Circularity Check
No significant circularity: the claims are empirical benchmark measurements, not derivations whose inputs entail the reported F1/ANLS drops.
full rationale
The paper's central claims (line-level and compound BBox+Pixel+Text attacks cause the largest degradation, up to 29.18% F1 drop; PGD beats random shift) are conclusions drawn from running attack procedures on standard test sets (FUNSD, CORD, SROIE, DocVQA) and measuring the resulting F1/ANLS changes. There is no equation-level derivation in which a target quantity is defined in terms of the attack or in which the prediction reduces by construction to its input. The layout attack maximizes L_adv = L_task - lambda_box(1 - IoU(B_hat, B_tilde)) in token-embedding space, and the reported performance drops are measured on the frozen VDU models after applying the resulting perturbed embeddings; the F1 drop is not used to construct the attack and is not an identity or a renamed fitting target. The bbox predictor g_theta is trained on the same datasets used for evaluation, which could inflate attack success if the predictor memorizes training boxes, but the paper does not define the evaluation metric in terms of g_theta, and the transferred-attack results (Tables 5-6) are measured on models other than the one used to generate the perturbations, so the central ranking claim is not forced by construction. The main scientific risk, noted in the reader's take, is that the PGD perturbation is computed on continuous token embeddings e~ with boxes recovered by g_theta(e~), and the paper does not demonstrate that e~ corresponds to a realizable document image whose OCR output reproduces it; this is a realizability gap in the threat model rather than a circularity in the derivation. No load-bearing self-citation chain is present, and no claim of a first-principles derivation is made for the robustness numbers. Accordingly, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- IoU budget tau =
{0.9, 0.75, 0.6}
- Text edit rate rho =
0.10
- PGD step count and step size =
T=10, alpha=0.05
- lambda_box =
not specified (only lambda_box > 0)
- lambda_GIoU =
2.0
assumptions (4)
- domain assumption Gradient-based optimization over token embeddings with a learned bbox predictor is a valid surrogate for physical layout edits of a document.
- domain assumption IoU between original and adversarially predicted boxes is a sufficient plausibility constraint for document layouts.
- domain assumption RoDLA's 12 transformations cover realistic pixel-level document perturbations.
- domain assumption Merging vertically aligned word boxes yields line-level annotations equivalent to real OCR line outputs.
Cite this review
Pith. "Pith review of Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks." pith.science (2026). https://pith.science/paper/MN3AQONR
@misc{pith2026250616407,
author = {Pith},
title = {Pith review of: Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/MN3AQONR}},
note = {Machine review of arXiv:2506.16407}
}
read the original abstract
Visual Document Understanding (VDU) systems have achieved strong performance in information extraction by integrating textual, layout, and visual signals. However, their robustness under realistic adversarial perturbations remains insufficiently explored. We introduce the first unified framework for generating and evaluating multi-modal adversarial attacks on OCR-based VDU models. Our method covers six gradient-based layout attack scenarios, incorporating manipulations of OCR bounding boxes, pixels, and texts across both word and line granularities, with constraints on layout perturbation budget (e.g., IoU >= 0.6) to preserve plausibility. Experimental results across four datasets (FUNSD, CORD, SROIE, DocVQA) and six model families demonstrate that line-level attacks and compound perturbations (BBox + Pixel + Text) yield the most severe performance degradation. Projected Gradient Descent (PGD)-based BBox perturbations outperform random-shift baselines in all investigated models. Ablation studies further validate the impact of layout budget, text modification, and adversarial transferability.
Figures
Reference graph
Works this paper leans on
-
[1]
Amazon Web Services . 2024. BoundingBox — Amazon Textract . https://docs.aws.amazon.com/textract/latest/dg/API_BoundingBox.html. Accessed: 2025-05-16
work page 2024
- [2]
-
[3]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923
arXiv 2025
-
[4]
Nicholas Boucher, Jenny Blessing, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2023. When Vision Fails : Text Attacks Against ViT and OCR
work page 2023
-
[5]
Yufan Chen, Jiaming Zhang, Kunyu Peng, Junwei Zheng, Ruiping Liu, Philip Torr, and Rainer Stiefelhagen. 2024 a . Rodla: Benchmarking the robustness of document layout analysis models. In CVPR
work page 2024
-
[6]
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198
2024
-
[7]
Google Cloud . 2024. Enterprise Document OCR — Google Cloud Document AI . https://cloud.google.com/document-ai/docs/form-parser. Accessed: 2025-05-16
work page 2024
-
[8]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...
arXiv 2024
Show all 30 references
-
[9]
Jiabang He, Yi Hu, Lei Wang, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. 2023. https://arxiv.org/abs/2306.02623 Do-good: Towards distribution shift evaluation for pre-trained visual document understanding models . Preprint, arXiv:2306.02623
2023 arXiv
-
[10]
Yupan Huang, Yiheng Xu, Lei Cui, and et al. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Findings of the Association for Computational Linguistics: EMNLP 2022
2022
-
[11]
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. https://doi.org/10.1109/icdar.2019.00244 Icdar2019 competition on scanned receipt ocr and information extraction . In 2019 International Conference on Document Analysis and...
2019
-
[12]
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In ICDAR
2019
-
[13]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge
2024
-
[14]
Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024 b . Safety of multimodal large language models on images and texts. arXiv preprint arXiv:2402.00357
2024 arXiv
-
[15]
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024 a . https://arxiv.org/abs/2403.05525 Deepseek-vl: Towards real-world vision-language underst...
2024 arXiv
-
[16]
Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, and 1 others. 2024 b . A bounding box is worth one token: Interleaving layout and text in a large language model for document understanding. arXiv preprint arXiv...
2024 arXiv
-
[17]
Chuwei Luo, Changxu Cheng, Qi Zheng, and Cong Yao. 2023. Geolayoutlm: Geometric pre-training for visual information extraction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
2023
-
[18]
Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
2021
-
[19]
Microsoft Azure . 2024. Read API — Azure AI Document Intelligence . https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout?view=doc-intel-4.0.0&tabs=rest Accessed: 2025-05-16
2024
-
[20]
Fatemeh Nourilenjan Nokabadi, Jean-Francois Lalonde, and Christian Gagné. 2024. https://arxiv.org/abs/2411.17468 Adversarial bounding boxes generation (abbg) attack against visual object trackers . Preprint, arXiv:2411.17468
2024 arXiv
-
[21]
OpenAI . 2024. Gpt-4.1-mini model overview. https://platform.openai.com/docs/models/gpt-4.1-mini. Accessed: 2025-05-20
2024
-
[22]
Seungjae Park, Seunghyun Shin, Bohyung Lee, and et al. 2019. Cord: A consolidated receipt dataset for post-ocr parsing. In Document Intelligence Workshop at NeurIPS
2019
-
[23]
Pham Phuc, Son Vuong, Khang Nguyen, and Tuan Dang. 2024. https://arxiv.org/abs/2412.18815 Distortion-aware adversarial attacks on bounding boxes of object detectors . Preprint, arXiv:2412.18815
2024 arXiv
-
[24]
Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2023. https://arxiv.org/abs/2306.13213 Visual adversarial examples jailbreak aligned large language models . Preprint, arXiv:2306.13213
2023 arXiv
-
[25]
Chenfan Qu, Chongyu Liu, Yuliang Liu, Xinhong Chen, Dezhi Peng, Fengjun Guo, and Lianwen Jin. 2023. https://api.semanticscholar.org/CorpusID:260072731 Towards robust tampered text detection in document image: New dataset and new solution . 2023 IEEE/CVF Conference on Computer ...
2023
-
[26]
Xijia Tao, Shuai Zhong, Lei Li, Qi Liu, and Lingpeng Kong. 2025. https://arxiv.org/abs/2403.02910 Imgtrojan: Jailbreaking vision-language models with one image . Preprint, arXiv:2403.02910
2025
-
[27]
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2022. https://arxiv.org/abs/2012.14740 Layoutlmv2: Multi-modal pre-training for visually-rich document understanding . Preprint, ar...
2022 arXiv
-
[28]
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. https://arxiv.org/abs/1905.07129 Ernie: Enhanced language representation with informative entities . Preprint, arXiv:1905.07129
2019 arXiv
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.