REVIEW 4 major objections 6 minor 24 references
DRISHTIKON: Visual Grounding at Multiple Granularities in Documents
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read DRISHTIKON claims that line-level localization, achieved by lexical matching between predicted answers and OCR regions, is the best operating point for document visual grounding, with a reported F1 of 69.10 on its MGVG benchmark.
desk verdict A useful new multilingual document grounding benchmark, but the headline granularity comparison is confounded by unequal aggregation limits and the 'SOTA' claim is a skyline under ground-truth answers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the hybrid region matching algorithm. For each OCR region it computes a composite score from a fuzzy text score (partial and token-based match between answer and region text), a length factor rewarding regions whose text length is plausible for the answer, and penalties for very short bounding boxes and for contextually unrelated text; regions above a threshold are kept, sorted, and returned down to a chosen top-k. This algorithm is what converts an answer string into block, line, word, and point boxes, and the ablations show that varying the number of allowed blocks and lines is what controls the reported precision-recall trade-off.
What would settle it
Corrupt each answer token in the MGVG OCR text by changing a single character (for example, 'transfer' to 'transfdr') and re-run the region matcher; the lexical-overlap mechanism predicts a large drop in line-level recall from 73.68, while a system that truly localizes semantically would degrade little. Conversely, if recall stays near its original value, the reported result is not actually driven by string matching.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a purely alignment-based matcher—no learned spatial grounding—is enough to localize extractive answers in structured, text-rich documents, provided the answer text is recoverable from OCR and the granularity is chosen to match the document's natural answer units. The paper reports that lines are that natural unit for government circulars: line-level grounding reaches an F1 of 69.10, above block (60.21), word (54.60), and point (55.79) levels. Allowing up to two blocks and around five lines improves recall without collapsing precision, and the same matcher outperforms vision-language models at both line and block level, with OCR-free models failing badly. The authors' stated conclusion is that mid-level granularity, especially line-level, should be the operating point for document VQA with grounding.
Load-bearing premise
The whole approach depends on the answer appearing almost verbatim in the OCR text, so that localizing it is a matter of string overlap; if OCR mangles the answer, splits or merges boxes unpredictably, or the question needs any inference beyond copying text, the matcher has no way to recover.
Editorial extensions
If this is right
- If line-level grounding is the right operating point, document VQA systems should emit line-level regions as their default provenance output and treat word/point output as a refinement for noise-free OCR only.
- Multi-line and multi-block answers require explicit aggregation parameters; the reported ablations give concrete starting values (about two blocks and five lines) for similar structured documents.
- Because an OCR-free vision-language model scores near zero on localization here, text-rich grounding systems should route through OCR plus alignment rather than relying on end-to-end visual features.
- The MGVG benchmark with human labels at four granularities can be reused as a testbed for grounding models and as a source of supervision for point-level finetuning.
Reading between the lines
- We infer the line-level optimum is genre-bound: in tables, forms, or multi-column pages the natural answer unit may be a cell or a field, so a re-run on such layouts could move the optimum away from lines.
- We infer the matcher could serve as a cheap annotation or distillation engine, generating coarse grounding labels that a vision-language model then refines, rather than as the end product; the paper's nod to point-level finetuning points this way.
- We infer the accuracy ceiling is set by OCR quality: any improvement in the matcher is capped by the fidelity of the transcribed text, so pairing the matcher with an OCR error-correction step should raise line-level recall beyond 73.68 on the same benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DRISHTIKON, a pipeline for multi-granular visual grounding in document images, combining multilingual OCR (DocTR, Surya), an LLM (LLaMA-3.1-8B) for answer prediction, and a heuristic region-matching algorithm that scores OCR blocks by fuzzy text similarity, length, and penalties. The authors introduce the MGVG benchmark: 70 Indian government circulars with 509 QnA pairs annotated at block, line, word, and point levels. Experiments report precision/recall/F1 across granularities, ablations over the maximum number of blocks and lines, and comparisons with LLaMA and Qwen2.5-VL. The paper's headline claim is that line-level grounding gives the best precision-recall balance (F1=69.10) and that the method is state-of-the-art.
Significance. If the benchmark and results are sound, the MGVG dataset is a useful resource for document VQA grounding evaluation, especially for multilingual, human-annotated administrative documents. The paper is transparent in labeling the ground-truth-answer evaluation as a skyline, and it makes code and dataset available. However, the central comparative claim is undermined by confounded hyperparameter settings and the lack of an end-to-end evaluation, as detailed below.
major comments (4)
- [Section 4.2, Tables 3, 6, 7] The block and line rows in Table 3 use different aggregation limits. The block row (P=53.89, R=68.22, F1=60.21) equals the max-blocks=5 row of Table 6, and the line row (P=65.06, R=73.68, F1=69.10) equals the max-lines=10 row of Table 7. Thus the granularity comparison does not hold the number of returned regions fixed. When both are capped at one region, Table 6 (max-blocks=1) gives F1=66.06 while Table 7 (max-lines=1) gives F1=56.19, reversing the conclusion. The claim that "line-level granularity provides the best balance between precision and recall" is therefore not supported by the experiments as presented. The paper's own text in Section 4.5 states that 5-line aggregation is optimal, yet the headline line numbers use 10 lines. This needs to be corrected with a matched-limit comparison or an explicit justification of why different aggregation limits are appropriate for each granularity.
- [Section 4.3 and Table 4] The headline F1=69.10 is obtained by feeding the ground-truth answer into the matcher; the paper labels this a "skyline" in the text, but the Abstract claims "state-of-the-art grounding accuracy" without this caveat. The only end-to-end line in Table 4 (predicted answer, LLaMA, Algorithm) has F1=48.12, which is well below the skyline and below the LLaMA-with-GT row. The state-of-the-art claim therefore applies to an oracle-conditioned component, not to the full DRISHTIKON pipeline. Please qualify the abstract and conclusion accordingly, or report end-to-end results as the headline.
- [Section 3.4 and Algorithm 1] The matching algorithm is underspecified. The composite score is described as a combination of fuzzy score, length factor, and penalties, but no equations, default weights, or threshold values are given (the text says "components which inturn have a scale of contribution" and "pre-defined threshold" without numbers). This prevents reproduction and makes it impossible to assess how sensitive the results are to these choices. In addition, the ablation in Section 4.5 selects the maximum number of blocks and lines from the test set (Tables 6 and 7) with no held-out validation split; the reported F1 values are therefore optimistically biased. Please specify the full scoring procedure and use a validation split (or cross-validation) for hyperparameter selection.
- [Section 4.2 and Tables 3-7] The paper reports no error bars, confidence intervals, or significance tests. Given the small dataset (70 documents, 509 QnA pairs) and the small differences in some ablations (e.g., F1 at 5 vs 10 lines: 69.33 vs 69.10), it is unclear which differences are meaningful. Please add bootstrapped confidence intervals or per-document variance, especially for the granularity comparison.
minor comments (6)
- [Throughout] The name is written inconsistently as "DRISHTIKON" and "DRISHTI KON"; please choose one spelling.
- [Section 4.5] The text contains a literal "Figure??" placeholder in "Table 7 and Figure?? show the variation"; please fix the cross-reference to Figure 6.
- [Table 4] The column headings of Table 4 ("Input (Text + BBox) OCR (QA) Model (QA) OCR (Grnd.) Model (Grnd.)") are confusing; the row entries use dashes and "YES"/"No" inconsistently, making it hard to determine which inputs each method receives.
- [Section 4.2] The word "qualititative" is misspelled in "Figure 4 illustrates the visual grounding qualititative results".
- [Section 2] Reference [3] is cited as a web page without a formal venue; if a peer-reviewed version or technical report is available, please cite it instead.
- [Table 2] Please clarify whether the annotation counts (e.g., 538 block regions, 5,968 word regions) are totals across the dataset or per-image averages; the caption is ambiguous.
Circularity Check
Headline grounding accuracy is an oracle upper bound: Table 3 feeds the ground-truth answer into a string-overlap matcher, and the line-vs-block comparison uses different test-set-tuned aggregation caps.
-
self definitional
[Algorithm 1 (Section 3.4) and Table 3 (Section 4.2)]
"The results, presented in Table 3, correspond to the predictions generated using our Region Matching Algorithm under the Question + Ground Truth Answer input. ... Algorithm 1: Input: ... 2. Predicted Answer Text A ... Step 1: Compute Fuzzy Score: a. Via partial match between A and text in r."
The region matcher scores every OCR block/line by textual overlap with A. When A is the ground-truth answer, the selected region is, by construction, the OCR region whose text contains that answer phrase, and the ground-truth annotation is the human-labeled region for that same phrase. The reported precision/recall therefore measures how well the OCR text contains the GT answer, not an independent prediction of the answer's location. The abstract's 'state-of-the-art grounding accuracy' thus reduces to an oracle string-match upper bound rather than a self-contained detection result.
-
fitted input called prediction
[Section 4.5, Tables 6-7, compared with Table 3]
"The optimal F1-score (62.68%) is achieved when allowing up to 2 blocks ... The F1-score stabilizes around 69.3%, with the optimal performance achieved at 5 lines. ... For our task, 2-block and 5-line aggregation offers the most balanced performance."
The max-blocks and max-lines caps are selected on the same test set that is later used to report the headline granularity results. Table 3's block F1=60.21 is exactly Table 6 at 5 blocks, while its line F1=69.10 is exactly Table 7 at 10 lines, so the granularity comparison confounds the aggregation cap with the granularity. Because the caps were tuned to maximize F1 on this test set, the 'line-level is best' conclusion is partly forced by the chosen evaluation configuration rather than by an independent property of the method. This is test-set fitting followed by a 'prediction' on the same data.
full rationale
The paper is not circular via self-citation: it cites no prior work by its own authors and imports no uniqueness theorem. The central circularity is in evaluation design. Algorithm 1 defines the output region as the OCR region with maximal fuzzy text overlap with an input answer string. When that string is the ground-truth answer (Tables 3 and 4, skyline rows), the reported F1 is a measure of OCR-and-annotation text overlap, not an independent learned prediction; the headline 'state-of-the-art grounding accuracy' is therefore an oracle result. In addition, the aggregation hyperparameters (max blocks, max lines) are chosen by ablations on the same test set used to produce Table 3, and the paper compares block-level at 5 candidate blocks against line-level at 10 candidate lines, so the claimed line-level advantage is confounded and optimistically biased. The full-pipeline numbers with LLaMA-predicted answers (F1=48.12 at line level) are considerably lower and are not the basis of the abstract's claim. No external benchmark is used, so there is no independent check that would break the cycle.
Assumptions & free parameters
free parameters (5)
- Composite score component weights =
Not stated
- Composite score threshold =
Not stated
- Maximum blocks (max_blocks) =
2 blocks selected as optimal in Table 6
- Maximum lines (max_lines) =
5 lines selected as optimal in Table 7; 10 lines used for headline Table 4
- Maximum boxes in evaluation (top-k) =
10 boxes in line-level evaluation (Table 4 caption)
assumptions (4)
- domain assumption DocTR and Surya OCR return accurate enough block/line text and bounding boxes on these circulars.
- domain assumption The answer text appears sufficiently verbatim in OCR content to be found by fuzzy/token matching.
- domain assumption Manual MGVG labels are correct and unambiguous.
- domain assumption IoU at 0.5 with up to 10 boxes is a fair protocol for measuring grounding quality.
Cite this review
Pith. "Pith review of DRISHTIKON: Visual Grounding at Multiple Granularities in Documents." pith.science (2026). https://pith.science/paper/6BC4KPK7
@misc{pith2026250621316,
author = {Pith},
title = {Pith review of: DRISHTIKON: Visual Grounding at Multiple Granularities in Documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/6BC4KPK7}},
note = {Machine review of arXiv:2506.21316}
}
read the original abstract
Visual grounding in text-rich document images is a critical yet underexplored challenge for Document Intelligence and Visual Question Answering (VQA) systems. We present DRISHTIKON, a multi-granular and multi-block visual grounding framework designed to enhance interpretability and trust in VQA for complex, multilingual documents. Our approach integrates multilingual OCR, large language models, and a novel region matching algorithm to localize answer spans at the block, line, word, and point levels. We introduce the Multi-Granular Visual Grounding (MGVG) benchmark, a curated test set of diverse circular notifications from various sectors, each manually annotated with fine-grained, human-verified labels across multiple granularities. Extensive experiments show that our method achieves state-of-the-art grounding accuracy, with line-level granularity providing the best balance between precision and recall. Ablation studies further highlight the benefits of multi-block and multi-line reasoning. Comparative evaluations reveal that leading vision-language models struggle with precise localization, underscoring the effectiveness of our structured, alignment-based approach. Our findings pave the way for more robust and interpretable document understanding systems in real-world, text-centric scenarios with multi-granular grounding support. Code and dataset are made available for future research.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Paddleocr, awesome multilingual ocr toolkits based on paddlepaddle
PaddlePaddle Authors. Paddleocr, awesome multilingual ocr toolkits based on paddlepaddle. https://github.com/ PaddlePaddle/PaddleOCR, 2020. 2
work page 2020
-
[3]
BharatGen Team. Bharatgen unveils patram: India’s pioneer- ing vision-language foundation model for document intelli- gence, 2025. 6
work page 2025
-
[4]
Molmo and pixmo: Open weights and open data for state- of-the-art vision-language models, 2024
Matt Deitke, Christopher Clark, Sangho Lee, and team. Molmo and pixmo: Open weights and open data for state- of-the-art vision-language models, 2024. 5
work page 2024
-
[5]
Abhimanyu Dubey and team. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. 2, 3, 5, 7
arXiv 2024
-
[6]
mplug- docowl2: High-resolution compressing for ocr-free multi- page document understanding
Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Mingshi Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug- docowl2: High-resolution compressing for ocr-free multi- page document understanding. ArXiv, abs/2409.03420,
-
[7]
Layoutlmv3: Pre-training for document ai with uni- fied text and image masking
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with uni- fied text and image masking. Proceedings of the 30th ACM International Conference on Multimedia, 2022. 2
work page 2022
-
[8]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023. 2
work page 2023
Show all 24 references
-
[9]
Towards visual text grounding of multimodal large language model
Ming Li, Ruiyi Zhang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tongfei Sun. Towards visual text grounding of multimodal large language model. ArXiv, abs/2504.04974, 2025. 2, 4
2025
-
[10]
Layoutllm: Layout instruction tun- ing with large language models for document understanding
Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tun- ing with large language models for document understanding. 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15630–15640, 2024. 2
2024
-
[11]
Manmatha, and C
Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V . Jawahar. Docvqa: A dataset for vqa on document im- ages. 2021 IEEE Winter Conference on Applications of Com- puter Vision (WACV), pages 2199–2208, 2020. 1
2021
-
[12]
doctr: Document text recognition
Mindee. doctr: Document text recognition. 2021. 2
2021
-
[13]
Nagaraja, Vlad I
Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis. Modeling context between objects for referring expression understanding. In European Conference on Computer Vi- sion, 2016. 1
2016
-
[14]
Surya: A lightweight document ocr and analysis toolkit
Vikas Paruchuri and Datalab Team. Surya: A lightweight document ocr and analysis toolkit. https://github. com/VikParuchuri/surya, 2025. GitHub repository. 2
2025
-
[15]
Go- ing full-tilt boogie on document understanding with text- image-layout transformer
Rafal Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Pałka. Go- ing full-tilt boogie on document understanding with text- image-layout transformer. ArXiv, abs/2102.09550, 2021. 2
2021 arXiv
-
[16]
Grounding of textual phrases in images by reconstruction
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Com- puter Vision, 2015. 1
2015
-
[17]
Unifying vision, text, and layout for universal document processing
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Chao-Yue Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 19...
2023
-
[18]
Gpt-4 technical report
OpenAI team. Gpt-4 technical report. 2023. 2
2023
-
[19]
Hierarchical multimodal transformers for multi-page docvqa
Rub `en P ´erez Tito, Dimosthenis Karatzas, and Ernest Val- veny. Hierarchical multimodal transformers for multi-page docvqa. ArXiv, abs/2212.05935, 2022. 1
2022 arXiv
-
[20]
Docllm: A layout-aware gener- ative language model for multimodal document understand- ing
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nour- bakhsh, and Xiaomo Liu. Docllm: A layout-aware gener- ative language model for multimodal document understand- ing. In Annual Meeting of the Association for Computat...
2023
-
[21]
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...
2024
-
[22]
Towards improving document understand- ing: An exploration on text-grounding via mllms
Yonghui Wang, Wen gang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. Towards improving document understand- ing: An exploration on text-grounding via mllms. ArXiv, abs/2311.13194, 2023. 2, 4, 5
2023 arXiv
-
[23]
Towards visual grounding: A survey
Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey. ArXiv, abs/2412.20206, 2024. 1
2024 arXiv
-
[24]
Doge: Towards versatile visual document grounding and referring
Yinan Zhou, Yuxin Chen, Haokun Lin, Shuyu Yang, Li Zhu, Zhongang Qi, Chen Ma, and Ying Shan. Doge: Towards versatile visual document grounding and referring. ArXiv, abs/2411.17125, 2024. 2, 4, 5
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.