{"id":"4cf1a4ff-5b18-4ee0-ac18-33ce608028c9","arxiv_id":"2412.00151","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"By feeding a montage of text crops with numbered boxes to a vision-language model, DLaVA localizes answers in documents and reports stronger accuracy and localization than the baselines it compares against.","lead":"DLaVA is a pipeline that makes a vision-language AI both answer questions about document images and point to exactly where each answer appears. The authors say it is training-free, cheaper to run than OCR-based systems, and more trustworthy because outputs are spatially grounded.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperforming previous SoTA' claim is not supported: Table 1's baselines lack a controlled prompt protocol and omit leading DocVQA systems, so the 5-point gain over Pixtral may be a prompt artifact.","rationale":"The reader's REJECT verdict is appropriate and I would not change it. The reader's weakest_assumption concerns missing ground-truth boxes for the IoU metric; I agree that is a serious unverifiability problem, particularly because Section 5.3's trustworthiness claim rests on those numbers. However, the single most load-bearing point for the central claim seems to me to be the uncontrolled textual-accuracy comparison: the SoTA headline depends on baseline prompts and a comparison pool that are neither specified nor complete. If that comparison is confounded, the paper's primary quantitative contribution is unsupported regardless of how the IoU protocol is eventually clarified. I therefore partially agree with the reader: I share the rejection but locate the decisive weakness one step earlier, at the ANLS comparison rather than the IoU protocol. The proposed controlled-prompt reproduction is cheap and would settle the point.","tokens_in":16659,"tokens_out":11499,"duration_ms":106226,"concrete_test":"Re-run Pixtral-12B (and at least Phi4-14B) on the same DocVQA split under three conditions: (1) the vendor/official prompt, (2) a direct-answer prompt with DLaVA's exact formatting instructions but without the constructed image and bounding-box inputs, and (3) DLaVA's full two-stage input. Also check the DocVQA public leaderboard for any published ANLS above 85.91 on the same split. If condition 2 closes most of the gap to condition 3, or if a published model exceeds 85.91, the SoTA claim is an artifact; if condition 3 is clearly best and 85.91 is genuinely above all published scores, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states that DLaVA 'achieves an ANLS score of 85.91%, outperforming previous SoTA approaches' on DocVQA, and the abstract extends this claim to six benchmarks. The evidence for this claim is underdetermined. First, Table 1 compares against a restricted set of MLLMs and layout-aware LLMs; leading specialized document-VQA systems on the public DocVQA ANLS leaderboard are absent, so 'previous SoTA' is not established as the field's SoTA. Second, the paper reports baseline numbers without the prompts or evaluation splits used to obtain them; Appendix C gives only the DLaVA prompt. That prompt imposes a strict constraint to answer 'only using words present in the image' and to return a formatted JSON with bounding boxes. ANLS rewards exact lexical matches, so such formatting can inflate measured accuracy independently of the constructed-image mechanism. Consequently, the 80.71-to-85.91 gap over Pixtral-12B could come from prompt engineering rather than from the proposed two-stage, constructed-image design. Because the SoTA textual-accuracy claim is the paper's central quantitative assertion, this is load-bearing: if the gap is a prompt artifact, the headline claim is unsupported even before the IoU protocol is examined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DLaVA, a training-free, two-stage pipeline for document VQA that augments an MLLM (Pixtral-12B) with a constructed image of detected text regions annotated by bounding-box IDs, plus an initial answer-extraction step. The authors claim state-of-the-art ANLS on several document benchmarks and report IoU-based spatial grounding metrics. The proposed method is evaluated on DocVQA, EST-VQA, RICO, FUNSD, CORD, and SROIE, with ablations comparing removal of the initial answer step, addition of the original image, and an OCR-dependent variant. The main contributions are framed as zero-shot answer localization, an OCR-free constructed-image design, and a dual ANLS+IoU evaluation protocol.","tokens_in":16862,"tokens_out":4474,"duration_ms":39173,"significance":"If the experimental claims were rigorously supported, the work would offer a simple, training-free mechanism for adding spatial grounding to an existing MLLM, and the emphasis on IoU alongside ANLS is a valuable direction for document VQA evaluation. The constructed-image idea is easy to understand and potentially reproducible, and the paper explicitly releases code and data. However, the current evaluation does not establish the central claims: baseline prompts are not disclosed, the IoU ground-truth protocol is underspecified for two datasets, and the 'SoTA' label is not supported by the selected comparison set. The core mechanism's contribution is also confounded with prompt effects. The significance is therefore conditional on substantial experimental revision.","major_comments":[{"comment":"The comparison with Pixtral-12B is uncontrolled. Appendix C gives the prompt used for DLaVA, but nothing is disclosed about the prompts used for the baseline MLLMs (Pixtral, Qwen2.5-VL, InternVL2, etc.). DLaVA's prompt imposes a strict 'only using words present in the image' constraint and a JSON output format, both of which can raise ANLS by forcing exact lexical matches. The reported 80.71→85.91 gain over Pixtral may therefore be due to prompt engineering rather than to the constructed-image two-stage pipeline. The authors must provide the exact prompts for all baselines and, crucially, a baseline that uses the same stage-1 prompt without the constructed-image stage, so the effect of the proposed mechanism can be isolated.","section":"Section 4.3 / Table 1 / Appendix C"},{"comment":"The IoU evaluation protocol is incomplete. DocVQA and RICO do not natively provide ground-truth bounding boxes for answer regions, yet Table 4 reports IoU (mAP@IoU[0.50:0.95]) for these datasets. The paper never states how the ground-truth boxes were generated for DocVQA and RICO. If they were obtained by running a text detector or by searching for the answer string, the IoU metric becomes circular and cannot support the claim of 'robust spatial grounding.' The authors must specify the annotation procedure, provide evidence that the boxes are human-approved or at least show the annotation interface and examples for every dataset, and discuss possible biases in the protocol.","section":"Section 4.1 / Table 4"},{"comment":"The claim that DLaVA 'outperforms previous SoTA approaches' on DocVQA is not substantiated. Table 1 compares only a small set of MLLMs and layout-aware LLMs; leading specialized document-VQA systems that top the public DocVQA ANLS leaderboard are absent. Furthermore, the abstract says 'competitive performance' while Section 5.1 says 'outperforming previous SoTA approaches,' which is internally inconsistent. Either the comparison set must be expanded to include the actual top-performing published methods on each benchmark, or the claim must be tempered to 'competitive with widely used open models' with the restricted comparison explicitly acknowledged.","section":"Section 5.1"},{"comment":"The contribution of the two-stage design to ANLS is confounded. In the proposed pipeline, the stage-1 answer text is passed unchanged to stage 2, and stage 2 only refines the bounding box. Therefore DLaVA's ANLS score is entirely determined by the stage-1 prompt. The comparison against plain Pixtral in Table 1 does not control for the prompt used, and Ablation 2 (removal of the information-extraction step) still uses the constructed image and bounding-box coordinates, so it does not isolate the stage-1 prompt's effect. The authors should add an ablation that uses only the stage-1 prompt (without the constructed image and without box IDs) to show that any ANLS gain over Pixtral is due to the pipeline and not the prompt wording.","section":"Section 3 / Table 3"},{"comment":"No error bars or multiple-run statistics are reported for any result, despite the use of stochastic sampling (temperature 0.1) in Pixtral. The reported margins over baselines are often around 5 ANLS points, and without variance estimates or at least multiple seeds, it is impossible to know whether these gaps are significant. The authors should report means and standard deviations over at least three runs for the main tables, or state if the metric is deterministic and justify that claim.","section":"Section 4.2 / Tables 1, 3, 4"}],"minor_comments":[{"comment":"The text states 76.74% for RICO, but Table 1 reports 76.34%; the discrepancy should be corrected.","section":"Section 5.1"},{"comment":"The spacing in 'DLaV A' is inconsistent; the model name should be written uniformly as 'DLaVA' (or 'DLaV-A') in all occurrences.","section":"Abstract and throughout"},{"comment":"RICO is a mobile-app UI dataset, not a document dataset; the paper should clarify how the document VQA task is defined on RICO and why it is appropriate as a document benchmark.","section":"Section 4.1"},{"comment":"The prompt specification is ambiguous: it says each 'bounding box' must contain exactly four numerical values, but the multi-word example contains a list of coordinate pairs per word. The format should be unified and explained more clearly.","section":"Appendix C"},{"comment":"The constructed-image definition in Section 3 uses 'Bn' for the final element while the figure and text use 'BBn'; the notation should be made consistent.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central ideas are simple and could be useful, but the current manuscript is far from providing the advertised evidence. The missing baseline prompts and undefined IoU ground truth for DocVQA/RICO are the most serious issues; they require rerunning experiments, not just rewriting. I would recommend the editor treat this as a major-revision decision with a strong expectation that the authors either provide the controlled comparisons and annotation protocol or substantially weaken their claims. The 'SoTA' language in Section 5.1 should be reconciled with the abstract's 'competitive' phrasing regardless of the experimental outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jana — quick read of DLaVA (2412.00151). The core idea is genuinely neat: use a text detector to crop every text region, assemble those crops into a single 'constructed image' with numbered IDs, then run a two-stage MLLM prompt — first answer from the original image, then localize on the constructed image using the ID coordinates. That is a new, practical variant of visual prompting; I do not recall seeing this exact combination. The ablations are also honest: they compared against an OCR-dependent variant and the constructed-image pipeline wins, which gives some support to the design.\n\nBut the paper's headline claims are not supported by the evidence. Three big problems.\n\nFirst, the SoTA label. Table 1 lists a handful of MLLMs and layout-aware LLMs, but omits leading specialized DocVQA systems (TAT-DQA, LayoutLMv3, Donut, UDOP, etc.). 'Previous SoTA' is not established as the field's SoTA. On the public DocVQA leaderboard, stronger numbers exist, so the 85.91 claim needs a much broader comparison set.\n\nSecond, the baseline prompts are not disclosed. Appendix C shows only the DLaVA prompt, which strictly forces answers to be words from the image and to be returned as JSON. ANLS rewards exact lexical matches, so this can inflate accuracy independently of the constructed-image mechanism. The 5-point gain over Pixtral could be a prompt artifact. For any serious evaluation, all baselines need the same prompt protocol, or a prompt-sweep.\n\nThird, the IoU metric is undefined for three of the five datasets. DocVQA and RICO do not natively provide answer-box annotations, and the paper never says how ground-truth boxes were generated. If they were obtained by searching for the answer string or running a detector, then the IoU scores are circular and the 'reduced hallucination' claim is unverifiable.\n\nMinor issues: no error bars, no sensitivity analysis of the hyperparameters, and no engagement with Set-of-Mark prompting, which is closely related. These are not fatal, but worth fixing.\n\nThat said, the method is simple, training-free, and plausibly useful. The problems are in the evaluation, not in the core idea. A rigorous resubmission with controlled prompting, full baselines, and a proper IoU protocol would be a solid paper.\n\nFor peer review: this deserves a serious referee, but my own verdict would be major revision, not acceptance as-is. The idea is worth one more cycle. I would not cite the numbers in the current form.","headline":"A practical training-free localization trick that is real, but the SoTA claims and IoU evaluation are too loose to support the headline numbers.","tokens_in":17494,"tokens_out":3463,"would_cite":false,"duration_ms":29317,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DLaVA claims a training-free, OCR-free pipeline that stitches detected text into one image with box IDs can beat previous document-VQA systems while also localizing the answers.","keywords":["document visual question answering","answer localization","multimodal large language model","zero-shot training-free","constructed image","OCR-free document understanding","spatial grounding","ANLS and IoU evaluation"],"falsifier":"Run DLaVA on DocVQA and RICO using independently human-annotated answer-region boxes and recompute the reported mAP over IoU thresholds; if the reference boxes used in the paper came from a text detector or an answer-string search, the localization claim will not survive this check even if the text answers remain correct.","tokens_in":16356,"feed_emoji":"📄","tokens_out":12947,"duration_ms":97803,"temperature":0.7,"pith_summary":"The paper sets out to show that document visual question answering can be done without OCR, without fine-tuning, and without chain-of-thought prompting: a multimodal large language model (MLLM) is asked twice, once to read the page and answer, and once to locate that answer given a single 'constructed image' made of detected text crops labeled with unique bounding-box IDs. It reports that this simple two-stage prompt beats specialized document models and prior OCR-free MLLMs on six benchmarks, including 85.91 ANLS on DocVQA, above the previous best, while also returning the screen position of every answer. That matters because giving users a box to check converts an unverifiable model output into evidence the reader can inspect, which is the paper's route to trustworthiness in legal, medical, and financial documents. The same design also shortens the prompt: instead of feeding all recognized text, the model sees crops and coordinates, which keeps long documents inside the context window and cuts computation.","feed_headline":"Training-free method tops DocVQA and gives answer boxes","feed_subtitle":"Stitching detected text crops into one image with box IDs lets a vision-language model answer and localize without OCR.","key_machinery":"The load-bearing object is the 'constructed image': a single synthetic image in which each detected text region is cropped and placed on its own line next to a unique identifier such as BB5, with the corresponding pixel coordinates supplied separately as a JSON list. This object replaces the OCR transcript. Because the MLLM can see the actual glyph shapes rather than a possibly errorful recognition string, the pipeline avoids cascading OCR errors; because all crops and IDs fit in one image, the prompt stays short enough for long documents; and because each answer token can be traced to an ID, the second MLLM call can output a bounding box instead of just a string. The two-stage prompting protocol is the second piece of machinery: Stage 1 extracts the textual answer, and Stage 2 converts it into a spatially grounded box using the constructed image and coordinate list.","core_discovery":"The central discovery claimed by the paper is that the spatial arrangement of a page can be preserved for an MLLM without OCR by assembling detected text-line crops into a constructed image in which every crop carries an ID and a coordinate list. The pipeline first runs a text detector (DB-ResNet-50) on the original page, crops each detected region, and lays the crops out as a new image with labels like BB1, BB2, alongside a JSON list of coordinates. Stage 1 feeds the original image and the user's question to Pixtral-12B and obtains a textual answer; Stage 2 feeds the constructed image, the coordinates, and the question-answer pair back to the same model and asks it to return the answer's bounding box. The paper reports that this two-stage, zero-shot design achieves an ANLS of 85.91% on DocVQA, 66.96% on EST-VQA, 76.34% on RICO, 87.57% on FUNSD, 84.41% on CORD, and 91.42% on SROIE, and that it outperforms ablation variants that add the original image, drop the first-stage extraction, or replace the constructed image with OCR text. It further claims the method provides spatial grounding measured by mean average precision over IoU thresholds from 0.50 to 0.95, with values of 46.22 on DocVQA, 33.65 on EST-VQA, 38.13 on RICO, 45.52 on FUNSD, and 57.86 on CORD, and presents those bounding boxes as the mechanism that makes the model's answers verifiable.","pith_inferences":["The IoU results should be read conditionally: since the paper does not say how reference boxes for DocVQA and RICO were produced, an independent human-annotation pass would be the natural way to confirm that the localization numbers measure genuine spatial understanding.","The constructed-image idea has an obvious next target beyond printed words: detected tables, embedded images, stamps, and signatures could be assigned IDs the same way, which would extend answer localization to non-text document content.","The paper's own CORD example, a perfect text match but a 0% IoU because repeated '11,000' values were confused, suggests the remaining error is semantic disambiguation rather than glyph recognition; injecting positional or cross-field context into the second-stage prompt is a cheap experiment to test that.","Because the pipeline is training-free, swapping the frozen MLLM while keeping the prompts fixed would reveal how much of the reported accuracy is the constructed-image design and how much is the underlying model's vision-language ability."],"forward_implications":["Document VQA can reach higher text accuracy with no training: the best configuration is a frozen MLLM plus a text detector, so upgrading either component does not require re-training.","Every answer is attached to a bounding box, so users can verify the response against the document; the paper presents this spatial grounding as the mechanism that reduces hallucinations.","Because no OCR transcript is sent to the model, dense documents no longer overflow the context window by token count; long inputs are compressed into one constructed image.","Reporting ANLS and IoU together means a system cannot claim success on its text alone; poor localization stays visible as a low IoU score.","The same training-free pipeline generalizes across English and Chinese questions (EST-VQA) and across receipts, forms, and mobile screens, so the method is not tied to one document genre."],"supporting_citations":[{"why":"Provides Pixtral-12B, the frozen MLLM used in both pipeline stages.","marker":"(Agrawal et al., 2024)"},{"why":"Provides DB-ResNet-50, the text detector whose crops and boxes form the constructed image.","marker":"(Liao et al., 2020)"},{"why":"Supplies DocVQA, the benchmark where the paper claims the 85.91 ANLS state-of-the-art result.","marker":"(Mathew et al., 2021)"},{"why":"Defines ANLS, the text-accuracy metric used for every comparison in the main table.","marker":"(Yujian & Bo, 2007)"},{"why":"Defines IoU and the mAP aggregation over IoU thresholds used for the spatial evaluation.","marker":"(Rezatofighi et al., 2019)"},{"why":"Supplies CORD, a receipt VIE benchmark the method is evaluated on and a source of ground-truth examples in the appendix.","marker":"(Park et al., 2019)"},{"why":"Supplies FUNSD, the form-understanding benchmark used for both ANLS and IoU evaluation.","marker":"(Jaume et al., 2019)"},{"why":"Provides LayoutLLM, a chain-of-thought layout baseline whose DocVQA number DLaVA claims to surpass.","marker":"(Luo et al., 2024)"},{"why":"Provides DocLayLLM, an OCR-dependent layout baseline that is the previous best on several reported benchmarks.","marker":"(Liao et al., 2024)"},{"why":"Supplies PARSeq, the text recognizer used in the OCR-dependent ablation variant.","marker":"(Bautista & Atienza, 2022)"}],"fun_headline_variants":["Training-free DLaVA marks answer boxes on docs","No OCR, no training: DLaVA boxes your answers","Zero-shot answer localization with DLaVA, no OCR","DLaVA: training-free, OCR-free answer pinpointing","Interpretable DocVQA answers with boxed locations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The spatial-accuracy scores stand on the assumption that every answer in the evaluated datasets has a trustworthy reference box to compare against; the paper never says where those boxes came from for DocVQA and RICO, so the localization numbers could reflect agreement with boxes that the system itself helped produce.","fun_headline_variants_meta":{"raw":{"variants":["Training-free DLaVA marks answer boxes on docs","No OCR, no training: DLaVA boxes your answers","Zero-shot answer localization with DLaVA, no OCR","DLaVA: training-free, OCR-free answer pinpointing","Interpretable DocVQA answers with boxed locations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3549,"prompt_tokens":1108,"completion_tokens":2441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":724,"completion_tokens_details":{"reasoning_tokens":2360}},"tokens_in":724,"tokens_out":2441,"duration_ms":16293,"temperature":1.0,"reasoning_tokens":2360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:07:28.111291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DLaVA on DocVQA and RICO using independently human-annotated answer-region boxes and recompute the reported mAP over IoU thresholds; if the reference boxes used in the paper came from a text detector or an answer-string search, the localization claim will not survive this check even if the text answers remain correct.","supporting_citations":[{"cited_title":"Real-time scene text detection with differentiable binarization","cited_arxiv_id":null,"evidence_quote":"Provides DB-ResNet-50, the text detector whose crops and boxes form the constructed image."},{"cited_title":"Generalized intersection over union: A metric and a loss for bounding box regression","cited_arxiv_id":null,"evidence_quote":"Defines IoU and the mAP aggregation over IoU thresholds used for the spatial evaluation."},{"cited_title":"Funsd: A dataset for form understanding in noisy scanned documents","cited_arxiv_id":null,"evidence_quote":"Supplies FUNSD, the form-understanding benchmark used for both ANLS and IoU evaluation."},{"cited_title":"Scene text recognition with permuted autoregressive sequence models","cited_arxiv_id":null,"evidence_quote":"Supplies PARSeq, the text recognizer used in the OCR-dependent ablation variant."}],"review_version":1}