{"id":"8a7e39a6-8037-4b99-a85c-bf3b4cffcb80","arxiv_id":"2506.21316","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A combined OCR, LLM, and fuzzy-matching pipeline locates answer spans in document images at block, line, word, and point granularity, with line-level grounding F1 of 69.10 on a new 70-document benchmark.","lead":"DRISHTIKON is a document-analysis system that finds where a question's answer appears in scanned government notices, down to the block, line, word, or point level. It combines OCR, a language model, and text-matching rules, and introduces a new manually annotated benchmark of 70 multilingual circulars with 509 question-answer pairs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Line-level superiority is confounded: Table 3 uses max-lines=10 and max-blocks=5, both test-set-tuned; at equal limits (1), block-level F1 (66.06) exceeds line-level (56.19).","rationale":"The reader's verdict of CONDITIONAL already flags test-set hyperparameter tuning and the lack of held-out validation. My concern sharpens this into a specific confound: the cross-granularity comparison in Table 3 uses different, test-set-selected aggregation limits for block-level (max-blocks=5) and line-level (max-lines=10), and the line-level numbers in the main tables do not match the paper's stated optimal of 5 lines. At an equal limit of 1, block-level F1 (66.06) is higher than line-level F1 (56.19), directly contradicting the claim that line-level is the best balance. This is a concrete, checkable issue that does not require new methods or external baselines—it can be settled by re-running the evaluation with controlled parameters. I therefore keep the reader's CONDITIONAL verdict unchanged, because the paper can address the concern by reporting the comparison at matched limits and on a validation split, but the central claim as currently presented is not adequately supported.","tokens_in":9030,"tokens_out":6772,"duration_ms":71543,"concrete_test":"Re-run the region matching evaluation on a held-out subset (e.g., 20 of the 70 documents) with the same maximum-region parameter (1, 2, 5, and 10) for both block-level and line-level grounding, using the exact scoring equations and thresholds (currently omitted from Algorithm 1). Report F1 at IoU=0.5 for each granularity at each limit. If line-level F1 does not exceed block-level F1 at every common limit, the 'line-level best' claim is parameter-dependent and must be revised or re-evaluated with a principled selection rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that line-level grounding offers the best precision-recall balance rests on a confounded comparison. The block-level results in Table 3 (P=53.89, R=68.22, F1=60.21) coincide exactly with the max-blocks=5 row of Table 6, and the line-level results (P=65.06, R=73.68, F1=69.10) coincide with the max-lines=10 row of Table 7. These are different aggregation limits, both selected from test-set ablations. The paper states that '2-block and 5-line aggregation offers the most balanced performance', yet the headline Table 3/Table 4 line-level numbers use max-lines=10, not the claimed optimum. When the aggregation limit is held equal across granularities, the conclusion reverses: Table 6 with max-blocks=1 gives F1=66.06, whereas Table 7 with max-lines=1 gives F1=56.19. Thus the apparent superiority of line-level grounding is an artifact of allowing up to ten candidate lines but only five candidate blocks. Since the hyperparameters are tuned on the test set with no validation split, the reported F1 values are optimistically biased, and the granularity comparison is not apples-to-apples. The claim as stated is therefore not supported by the experiments as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DRISHTIKON, a pipeline for multi-granular visual grounding in document images, combining multilingual OCR (DocTR, Surya), an LLM (LLaMA-3.1-8B) for answer prediction, and a heuristic region-matching algorithm that scores OCR blocks by fuzzy text similarity, length, and penalties. The authors introduce the MGVG benchmark: 70 Indian government circulars with 509 QnA pairs annotated at block, line, word, and point levels. Experiments report precision/recall/F1 across granularities, ablations over the maximum number of blocks and lines, and comparisons with LLaMA and Qwen2.5-VL. The paper's headline claim is that line-level grounding gives the best precision-recall balance (F1=69.10) and that the method is state-of-the-art.","tokens_in":9315,"tokens_out":5737,"duration_ms":54840,"significance":"If the benchmark and results are sound, the MGVG dataset is a useful resource for document VQA grounding evaluation, especially for multilingual, human-annotated administrative documents. The paper is transparent in labeling the ground-truth-answer evaluation as a skyline, and it makes code and dataset available. However, the central comparative claim is undermined by confounded hyperparameter settings and the lack of an end-to-end evaluation, as detailed below.","major_comments":[{"comment":"The block and line rows in Table 3 use different aggregation limits. The block row (P=53.89, R=68.22, F1=60.21) equals the max-blocks=5 row of Table 6, and the line row (P=65.06, R=73.68, F1=69.10) equals the max-lines=10 row of Table 7. Thus the granularity comparison does not hold the number of returned regions fixed. When both are capped at one region, Table 6 (max-blocks=1) gives F1=66.06 while Table 7 (max-lines=1) gives F1=56.19, reversing the conclusion. The claim that \"line-level granularity provides the best balance between precision and recall\" is therefore not supported by the experiments as presented. The paper's own text in Section 4.5 states that 5-line aggregation is optimal, yet the headline line numbers use 10 lines. This needs to be corrected with a matched-limit comparison or an explicit justification of why different aggregation limits are appropriate for each granularity.","section":"Section 4.2, Tables 3, 6, 7"},{"comment":"The headline F1=69.10 is obtained by feeding the ground-truth answer into the matcher; the paper labels this a \"skyline\" in the text, but the Abstract claims \"state-of-the-art grounding accuracy\" without this caveat. The only end-to-end line in Table 4 (predicted answer, LLaMA, Algorithm) has F1=48.12, which is well below the skyline and below the LLaMA-with-GT row. The state-of-the-art claim therefore applies to an oracle-conditioned component, not to the full DRISHTIKON pipeline. Please qualify the abstract and conclusion accordingly, or report end-to-end results as the headline.","section":"Section 4.3 and Table 4"},{"comment":"The matching algorithm is underspecified. The composite score is described as a combination of fuzzy score, length factor, and penalties, but no equations, default weights, or threshold values are given (the text says \"components which inturn have a scale of contribution\" and \"pre-defined threshold\" without numbers). This prevents reproduction and makes it impossible to assess how sensitive the results are to these choices. In addition, the ablation in Section 4.5 selects the maximum number of blocks and lines from the test set (Tables 6 and 7) with no held-out validation split; the reported F1 values are therefore optimistically biased. Please specify the full scoring procedure and use a validation split (or cross-validation) for hyperparameter selection.","section":"Section 3.4 and Algorithm 1"},{"comment":"The paper reports no error bars, confidence intervals, or significance tests. Given the small dataset (70 documents, 509 QnA pairs) and the small differences in some ablations (e.g., F1 at 5 vs 10 lines: 69.33 vs 69.10), it is unclear which differences are meaningful. Please add bootstrapped confidence intervals or per-document variance, especially for the granularity comparison.","section":"Section 4.2 and Tables 3-7"}],"minor_comments":[{"comment":"The name is written inconsistently as \"DRISHTIKON\" and \"DRISHTI KON\"; please choose one spelling.","section":"Throughout"},{"comment":"The text contains a literal \"Figure??\" placeholder in \"Table 7 and Figure?? show the variation\"; please fix the cross-reference to Figure 6.","section":"Section 4.5"},{"comment":"The column headings of Table 4 (\"Input (Text + BBox) OCR (QA) Model (QA) OCR (Grnd.) Model (Grnd.)\") are confusing; the row entries use dashes and \"YES\"/\"No\" inconsistently, making it hard to determine which inputs each method receives.","section":"Table 4"},{"comment":"The word \"qualititative\" is misspelled in \"Figure 4 illustrates the visual grounding qualititative results\".","section":"Section 4.2"},{"comment":"Reference [3] is cited as a web page without a formal venue; if a peer-reviewed version or technical report is available, please cite it instead.","section":"Section 2"},{"comment":"Please clarify whether the annotation counts (e.g., 538 block regions, 5,968 word regions) are totals across the dataset or per-image averages; the caption is ambiguous.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The MGVG benchmark is a small but potentially useful resource. The main issues are methodological: the granularity comparison is confounded, the skyline evaluation is presented as the headline result, and the algorithm is underspecified. These are fixable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MGVG benchmark is the real contribution here. Seventy real multilingual circulars with 509 human-annotated QA pairs at four granularities is a legitimate asset, especially because the prior benchmarks they cite still aren't released. That alone makes the paper worth a look.\n\nThe problem is the headline. The claim that line-level grounding gives the best precision-recall balance is not supported by their own tables. Table 3's line-level F1 of 69.10 comes from the max-lines=10 row of Table 7, while the block-level F1 of 60.21 comes from the max-blocks=5 row of Table 6. Those are different aggregation limits, both selected from test-set ablations. When you hold the limit equal, as the stress-test note does, the conclusion reverses: max-blocks=1 gives F1 66.06, max-lines=1 gives 56.19. So the comparison is apples-to-oranges, and the paper's own stated optimum of \"2-block and 5-line aggregation\" doesn't match the 10-line setting used in the main result. That's a real inconsistency, not a nitpick.\n\nThe second issue is that the headline F1 is a skyline. Table 4 explicitly labels the ground-truth-answer condition as \"skyline performance,\" but the abstract says \"achieves state-of-the-art grounding accuracy\" without that caveat. The system needs the answer text to already exist nearly verbatim in the OCR output, so the matcher is doing lexical overlap, not independent grounding. That's fine as an upper bound, but it's not a SOTA system claim.\n\nThe matching algorithm itself is also underspecified. Fuzzy score, length factor, penalties, threshold, and the weights that combine them appear only as prose and a pseudocode sketch. Combined with hyperparameters tuned directly on the test set and no error bars, the quantitative results should be read as optimistic.\n\nWhere the paper does earn credit: the ablations in Tables 6 and 7 do show a real precision-recall trade-off with aggregation, and the qualitative comparison of algorithm versus LLM grounding is honest about both failing in different ways. The benchmark, with its manual annotation and multilingual coverage, is something I'd want to see preserved even if the method section needs rework.\n\nBottom line: the benchmark deserves serious referee time, but the central empirical claim needs to be re-run with matched aggregation limits, a proper validation split, and the algorithm fully specified. I'd recommend conditional acceptance after major revision, not desk rejection and not acceptance as is.","headline":"A useful new multilingual document grounding benchmark, but the headline granularity comparison is confounded by unequal aggregation limits and the 'SOTA' claim is a skyline under ground-truth answers.","tokens_in":9859,"tokens_out":1722,"would_cite":true,"duration_ms":20394,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DRISHTIKON claims that line-level localization, achieved by lexical matching between predicted answers and OCR regions, is the best operating point for document visual grounding, with a reported F1 of 69.10 on its MGVG benchmark.","keywords":["document visual grounding","multi-granular localization","document VQA","region matching","OCR","large language model","MGVG benchmark","interpretability"],"falsifier":"Corrupt each answer token in the MGVG OCR text by changing a single character (for example, 'transfer' to 'transfdr') and re-run the region matcher; the lexical-overlap mechanism predicts a large drop in line-level recall from 73.68, while a system that truly localizes semantically would degrade little. Conversely, if recall stays near its original value, the reported result is not actually driven by string matching.","tokens_in":8820,"feed_emoji":"📄","tokens_out":7788,"duration_ms":86296,"temperature":0.7,"pith_summary":"DRISHTIKON is a pipeline for visual grounding in document images: multilingual OCR splits the page into located text blocks, a large language model reads the OCR text and produces an extractive answer, and a region matching algorithm scores every block, line, and word by fuzzy textual overlap with that answer, returning the strongest regions plus point locations. The paper introduces the Multi-Granular Visual Grounding (MGVG) test set of 70 Indian government circulars with 509 human-annotated question-answer pairs labeled at block, line, word, and point granularity. Its central claim is that line-level localization gives the best precision-recall balance, reaching 65.06 precision, 73.68 recall, and 69.10 F1 with ground-truth answers and OCR boxes, and that this beats leading vision-language models tested under the same conditions. The value of the claim is that it makes answer provenance in text-heavy documents a verifiable output rather than a byproduct.","feed_headline":"Line-level grounding beats word and block on document VQA: F1 69.10","feed_subtitle":"DRISHTIKON's OCR-plus-matching pipeline localizes answer spans at four granularities; line level balances precision and recall.","key_machinery":"The load-bearing object is the hybrid region matching algorithm. For each OCR region it computes a composite score from a fuzzy text score (partial and token-based match between answer and region text), a length factor rewarding regions whose text length is plausible for the answer, and penalties for very short bounding boxes and for contextually unrelated text; regions above a threshold are kept, sorted, and returned down to a chosen top-k. This algorithm is what converts an answer string into block, line, word, and point boxes, and the ablations show that varying the number of allowed blocks and lines is what controls the reported precision-recall trade-off.","core_discovery":"On the paper's own terms, the discovery is that a purely alignment-based matcher—no learned spatial grounding—is enough to localize extractive answers in structured, text-rich documents, provided the answer text is recoverable from OCR and the granularity is chosen to match the document's natural answer units. The paper reports that lines are that natural unit for government circulars: line-level grounding reaches an F1 of 69.10, above block (60.21), word (54.60), and point (55.79) levels. Allowing up to two blocks and around five lines improves recall without collapsing precision, and the same matcher outperforms vision-language models at both line and block level, with OCR-free models failing badly. The authors' stated conclusion is that mid-level granularity, especially line-level, should be the operating point for document VQA with grounding.","pith_inferences":["We infer the line-level optimum is genre-bound: in tables, forms, or multi-column pages the natural answer unit may be a cell or a field, so a re-run on such layouts could move the optimum away from lines.","We infer the matcher could serve as a cheap annotation or distillation engine, generating coarse grounding labels that a vision-language model then refines, rather than as the end product; the paper's nod to point-level finetuning points this way.","We infer the accuracy ceiling is set by OCR quality: any improvement in the matcher is capped by the fidelity of the transcribed text, so pairing the matcher with an OCR error-correction step should raise line-level recall beyond 73.68 on the same benchmark."],"forward_implications":["If line-level grounding is the right operating point, document VQA systems should emit line-level regions as their default provenance output and treat word/point output as a refinement for noise-free OCR only.","Multi-line and multi-block answers require explicit aggregation parameters; the reported ablations give concrete starting values (about two blocks and five lines) for similar structured documents.","Because an OCR-free vision-language model scores near zero on localization here, text-rich grounding systems should route through OCR plus alignment rather than relying on end-to-end visual features.","The MGVG benchmark with human labels at four granularities can be reused as a testbed for grounding models and as a source of supervision for point-level finetuning."],"supporting_citations":[{"why":"Supplies the large language model that reads OCR text and produces the predicted answers the matcher localizes; also serves as an LLM-based grounding baseline.","marker":"[5]"},{"why":"Provides the primary multilingual OCR with layout prediction that produces the block-level regions and bounding boxes used throughout the pipeline.","marker":"[12]"},{"why":"Provides a second multilingual OCR engine used alongside the primary one to improve text extraction in the document images.","marker":"[14]"},{"why":"Acts as the vision-language-model baseline whose near-zero localization scores support the paper's claim that OCR-free VLMs cannot do this task.","marker":"[21]"},{"why":"A prior text-grounding dataset the paper compares against and cites as unreleased, used to justify why a new human-annotated benchmark is needed.","marker":"[22]"},{"why":"A multi-granular grounding benchmark whose synthetic construction is contrasted with the manual MGVG annotations; central to the comparison table.","marker":"[24]"},{"why":"A layout-heavy grounding benchmark used in the same comparison to position MGVG against existing real-world test sets.","marker":"[9]"}],"fun_headline_variants":["Line-level grounding tops word and block in document VQA: F1 69.10","DRISHTIKON: line granularity best for document grounding, F1 69.10","Alignment-based matcher wins doc grounding at line level, F1 69.10","Document VQA grounding: lines beat words and blocks, F1 69.10","For document VQA, line-level localization hits F1 69.10, best of four"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach depends on the answer appearing almost verbatim in the OCR text, so that localizing it is a matter of string overlap; if OCR mangles the answer, splits or merges boxes unpredictably, or the question needs any inference beyond copying text, the matcher has no way to recover.","fun_headline_variants_meta":{"raw":{"variants":["Line-level grounding tops word and block in document VQA: F1 69.10","DRISHTIKON: line granularity best for document grounding, F1 69.10","Alignment-based matcher wins doc grounding at line level, F1 69.10","Document VQA grounding: lines beat words and blocks, F1 69.10","For document VQA, line-level localization hits F1 69.10, best of four"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1541,"prompt_tokens":955,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":571,"tokens_out":586,"duration_ms":6314,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:27:34.969254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt each answer token in the MGVG OCR text by changing a single character (for example, 'transfer' to 'transfdr') and re-run the region matcher; the lexical-overlap mechanism predicts a large drop in line-level recall from 73.68, while a system that truly localizes semantically would degrade little. Conversely, if recall stays near its original value, the reported result is not actually driven by string matching.","supporting_citations":[{"cited_title":"doctr: Document text recognition","cited_arxiv_id":null,"evidence_quote":"Provides the primary multilingual OCR with layout prediction that produces the block-level regions and bounding boxes used throughout the pipeline."},{"cited_title":"Surya: A lightweight document ocr and analysis toolkit","cited_arxiv_id":null,"evidence_quote":"Provides a second multilingual OCR engine used alongside the primary one to improve text extraction in the document images."},{"cited_title":"Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024","cited_arxiv_id":null,"evidence_quote":"Acts as the vision-language-model baseline whose near-zero localization scores support the paper's claim that OCR-free VLMs cannot do this task."}],"review_version":1}