{"id":"8f1f0055-4859-462f-a539-104c23d1079f","arxiv_id":"2411.17125","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors build a data-generation engine, a seven-task grounding/referring benchmark, and a model that localizes and reads text in document images better than existing MLLMs on that benchmark.","lead":"This paper introduces DOGR-Engine, a pipeline that generates millions of fine-grained text grounding annotations for posters, charts, and PDFs, plus DOGR-Bench, a seven-task benchmark for measuring that skill. It also trains DOGR, a multimodal language model that grounds and refers to text in documents, and reports strong results on the new benchmark and competitive results on standard document tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DOGR-Bench may overlap DOGR's training set because both come from DOGR-Engine; without a reported train/test split or overlap audit, the Tab. 2 performance gap is not yet interpretable as a capability difference.","rationale":"I read the paper in good faith: the contributions are the data engine, the benchmark, and the model, and there is independent evidence of value in the DocLocal4K results, the 10 general-document benchmarks, and the ablations in Tabs. 4-5 and Appendix G. However, the strongest claim is specifically about beating all existing SOTA models on DOGR-Bench, and that claim depends entirely on DOGR-Bench being a clean evaluation. The reader's weakest assumption is exactly the load-bearing one: both the training data and the benchmark are produced by the same engine from the same source corpora, and the paper never reports a train/test split or overlap analysis. Section 3.3 is titled 'Data Verification and Splitting' but its text covers only filtering and correction; Section 4 gives benchmark statistics but no provenance or deduplication details relative to training. This gap is not proof that leakage occurred, but it is a concrete, checkable condition for the central claim, and the size of the reported performance difference makes the audit essential. A public overlap computation and a disjoint-subset rerun would settle the question. The existing conditional verdict is therefore appropriate; I do not see grounds to move it to accept or reject based on the current manuscript.","tokens_in":859,"tokens_out":777,"duration_ms":27614,"concrete_test":"Release or compute an overlap audit: hash all DOGR-Bench images and all DOGR training images with a perceptual hash (e.g., pHash/dHash) at a high threshold, and also perform exact/near-duplicate matching on normalized '<ocr>/<bbox>' strings after stripping whitespace; report the fraction of benchmark samples whose image or QA pair appears in the training set. Then rerun Tab. 2 on the disjoint subset only. If overlap is 0 and the headline scores on the disjoint subset reproduce within roughly 2 points, the central comparison stands; if overlap exceeds about 5% of any task, the headline claim should be revised or the benchmark rebuilt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that DOGR 'substantially outperforms all existing open-source and proprietary SOTA models' (Sec. 6.2, Tab. 2), requires DOGR-Bench to be a fair, uncontaminated test. The 3.6K benchmark samples are generated by the same DOGR-Engine, from the same poster (Crello), chart (ChartQA), and PDF (CC-MAIN) sources, and in the same GPT-4o '<ocr>/<bbox>' format as the 700K instruction-tuning samples. Yet Sec. 3.3, despite its title 'Data Verification and Splitting', describes only rule-based filtering and correction; no train/test split, deduplication, or overlap analysis is reported in Sec. 4 or Appendix D. If benchmark images or grounded QA pairs coincide with fine-tuning data, the 76.3 vs 38.8 Ga F1all gap in Tab. 2 is largely a memorization artifact. This is not an accusation of bad faith; it is a missing audit that the paper itself should supply, especially since the release includes code, data, and model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DOGR-Engine, a data generation pipeline that produces multi-granular parsing data (2.1M samples) and instruction-tuning data (700K samples) for document grounding and referring tasks, and DOGR-Bench, a 3.6K-sample benchmark covering seven ground-and-refer tasks across posters, charts, and PDF documents. The authors also train DOGR, an MLLM that integrates text grounding and referring into dialogue and reasoning, and report that it substantially outperforms existing open-source and proprietary MLLMs on DOGR-Bench while remaining competitive on ten traditional document understanding benchmarks. The paper claims as contributions the data engine, the benchmark, and the model.","tokens_in":21265,"tokens_out":5773,"duration_ms":53217,"significance":"If the central evaluation is valid, this is a valuable contribution: the data engine addresses the scarcity of high-quality, diverse document ground-and-refer data; the benchmark provides a clear task taxonomy and quantitative evaluation for an underdeveloped capability; and the model demonstrates that the generated data can improve a strong baseline. The release of code, data, and model is a tangible asset to the community. The main caveat is that the benchmark and the training data are generated by the same engine, and the paper does not establish that the benchmark is free of overlap with the training set, which is load-bearing for the headline claims.","major_comments":[{"comment":"The paper does not report any train/test overlap control between DOGR-Bench and the DOGR training data. Both are constructed by DOGR-Engine from the same sources (Crello posters, ChartQA charts, CC-MAIN PDFs) and with the same GPT-4o '<ocr>/<bbox>' annotation format. Section 3.3, despite its title 'Data Verification and Splitting', describes only rule-based filtering and correction; no splitting, deduplication, or overlap analysis is reported in Sec. 4 or Appendix D. Without such an audit, the large gap in Tab. 2 (e.g., 76.3 vs 38.8 Ga F1all at IoU 0.1) cannot be interpreted as a capability difference rather than an in-distribution memorization advantage. This is load-bearing for the central claim in Sec. 6.2 that DOGR 'substantially outperforms all existing open-source and proprietary SOTA models.' I request an explicit overlap analysis (e.g., image hashing and exact/near-duplicate matching of grounded text-bbox pairs) and, if any overlap exists, a re-evaluation on a disjoint subset.","section":"Sec. 3.3, Sec. 4, Tab. 2"},{"comment":"The headline comparison in Tab. 2 pits DOGR, which is fine-tuned on 700K ground-and-refer instruction samples produced by DOGR-Engine, against general-purpose MLLMs that have not been trained for document grounding-and-referring. The claim that DOGR 'substantially outperforms all existing open-source and proprietary SOTA models' is therefore misleading without a document-specialized baseline trained on comparable grounding data; it would be more accurate to state that a model fine-tuned on DOGR-Engine data outperforms generalist baselines on this benchmark. The authors should either add at least one such specialized baseline (e.g., a variant of DocOwl-1.5 or Kosmos-2.5 fine-tuned with grounding instruction data) or temper the wording of the claim.","section":"Sec. 6.2, Tab. 2"}],"minor_comments":[{"comment":"The section title 'Data Verification and Splitting' is misleading because no actual splitting procedure is described; either add the split details or rename the section to 'Data Verification and Filtering'.","section":"Sec. 3.3"},{"comment":"The column header 'BLEU4F1all' is ambiguous; it should be split into two separate headers ('BLEU4' and 'F1all') for clarity.","section":"Tab. 2"},{"comment":"There is a typo in Appendix B.1: 'sampls' should be 'samples'.","section":"Appendix B.1"},{"comment":"In the text above Tab. 10, 'DO-Bench' should be 'DOGR-Bench'.","section":"Appendix G, Tab. 10"},{"comment":"The paper reports single runs without error bars or significance tests. While the main conclusions are unlikely to change given the large gaps, reporting multiple seeds or confidence intervals would be useful, especially for subset-level results (e.g., the per-document-type rows in Tab. 2).","section":"All experimental tables"},{"comment":"The claim of being the 'first comprehensive benchmark' for document grounding and referring should be carefully worded, since some existing benchmarks include related region-level tasks; the novelty should be positioned relative to prior work in the related work section.","section":"Sec. 1 and Sec. 7"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the potential contamination between DOGR-Bench and the training data, which is directly tied to the paper's main claim. This is fixable in a revision by providing an overlap analysis and re-computing results on a clean split. The paper would then be a solid contribution to the document understanding community. I would also encourage the authors to include a specialized document-grounding baseline in the comparison, as the current comparison against generalist models overstates the model's novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful resource paper with a real, fixable flaw. The data-engine tricks are clever, but the central benchmark claim is currently uninterpretable because DOGR-Bench and DOGR's training set come from the same engine and no overlap audit is reported.\n\nWhat's actually new and good: the re-rendering strategy for posters and charts is a clean way to get exact bounding boxes without OCR noise; the MinerU + PyMuPDF merge for PDF reading order is pragmatic and well explained; the seven-task taxonomy (Ga, Gr, Go, GRa, GRr, GRo, Rt) gives a useful structure for document grounding evaluation. The paper ships code, data, and model, and the ablation studies (Tabs. 4, 5, 8–10) show real gains on external benchmarks like DocLocal4K, ChartQA, and VisualMRC. That external validation matters.\n\nThe soft spot is exactly where the reader put it. Section 3.3 is titled 'Data Verification and Splitting,' but the text only describes rule-based filtering and categorization. There is no train/test split, deduplication, or overlap analysis between DOGR-Bench and the 700K instruction-tuning samples, even though both are produced by DOGR-Engine from the same poster/chart/PDF sources in the same GPT-4o format. Without that audit, the 76.3 vs 38.8 gap in Tab. 2 could be largely in-distribution memorization. This is a missing audit, not an accusation of bad faith, but it is load-bearing: the paper's headline claim rests on it.\n\nSmaller issues: single runs with no error bars, and baselines are necessarily prompted differently, both of which weaken the comparisons. The 'first comprehensive benchmark' phrasing is also a bit strong, though I don't know a prior benchmark that covers exactly this task set.\n\nThe paper does not rely solely on DOGR-Bench: performance on DocLocal4K and ten general benchmarks, plus the ablations, indicates the data engine has value independent of the benchmark. The right fix is to release an overlap audit and, if needed, re-split the benchmark so it is genuinely held out. If the gap persists after that, the claim is strong. As written, the benchmark comparison is conditional.\n\nThis is a paper for people building fine-grained document assistants and MLLM data pipelines. A serious referee should engage with it; the flaw is addressable and the resource contribution is real. I would accept it for review, with a request for the overlap analysis as a major revision.","headline":"Useful data-engine paper with a real, fixable flaw: the headline DOGR-Bench gap is uninterpretable without an overlap audit between the benchmark and the training data.","tokens_in":21795,"tokens_out":3447,"would_cite":true,"duration_ms":29357,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By training on precisely grounded document parsing and instruction data, the paper builds a model that can point to the exact words it reads and shows that existing multimodal systems lag far behind on this ability.","keywords":["document grounding","referring","multimodal large language model","visual document understanding","data engine","benchmark","bounding box annotation","instruction tuning"],"falsifier":"Apply a deduplication check that compares every DOGR-Bench test image (or a re-rendered variant) and every test question-answer pair against DOGR's 703K instruction samples and 2.1M parsing samples; if a substantial fraction of test documents or grounded QA pairs appear verbatim or near-verbatim in the fine-tuning data, the headline grounding gap would be explained by memorization rather than by the data engine's claimed capability gain.","tokens_in":3,"feed_emoji":"🎯","tokens_out":7482,"duration_ms":125346,"temperature":0.7,"pith_summary":"The paper seeks to establish that fine-grained document understanding—reading text and knowing exactly which region of the page that text occupies—can be trained systematically with automatically generated data, and that current multimodal systems are far behind on this skill. It introduces DOGR-Engine, a pipeline that builds precise multi-granular parsing annotations (word, phrase, line, paragraph, full page) and 703K instruction samples that teach a model to ground its answers in specific text and to refer to user-selected regions. On the resulting DOGR-Bench, which spans seven grounding and referring tasks over charts, posters, and PDFs, the trained model DOGR reaches 76.3 F1all on the base grounding task at a relaxed IoU, against 38.8 for the strongest existing system, and it keeps a large margin at the strict IoU of 0.5. The authors' central claim is that the missing ingredient for grounded document interaction is high-quality generated data, not a new architecture.","feed_headline":"Beats every prior model at pointing to text in documents","feed_subtitle":"A data engine writes 703K grounded QA pairs; a 7-task benchmark shows a 38-point gap over existing AI.","key_machinery":"The load-bearing mechanism is the DOGR-Engine's annotation pipeline. For posters and charts it re-renders the document once with a single text block's color changed and subtracts the two images, isolating that block's exact bounding box; for PDFs it merges a reading-order-aware extraction with a comprehensive text extractor so that every block is both present and ordered. Instruction samples are produced by asking a large multimodal model to answer questions about these images while wrapping every quoted document text in <ocr>...</ocr> markers, after which the engine retrieves coordinates from the PDF parser and attaches them as <bbox>...</bbox> tags. This results in 2.1M parsing samples and 703K instruction samples whose grounding annotations are spatial and text-consistent, which is what trains the model to couple location and content rather than read one without the other.","core_discovery":"The paper's central discovery is that a generic vision-encoder-plus-LLM architecture becomes capable of document grounding and referring once trained on parsing and instruction data that consistently couple text with precise normalized coordinates. The data engine produces these couplings by re-deriving bounding boxes from the rendering process itself rather than trusting OCR outputs, and by asking a strong API model to generate queries and responses in a grounded text-and-box format. On the paper's own benchmark, the resulting model's grounding score (Ga F1all at IoU 0.1) is 76.3 versus 38.8 for the best of eight existing models, and its referring accuracy (Rt) is 60.3 versus 43.0 for the best baseline. The same model is also competitive on ten standard document understanding tasks, indicating the grounding capability comes without sacrificing reading comprehension.","pith_inferences":["If the data engine is the true cause of the leap, then applying the same pipeline to other base MLLMs should substantially raise their grounding scores; the paper does not report such a controlled transfer experiment.","The re-rendering strategy generalizes in principle to any template-based document source with re-renderable text, so the engine could extend to slides, forms, or web layouts with little change.","Because both DOGR-Bench and DOGR's training set originate from the same documents and the same question-generation scheme, an independent train/test overlap check would determine whether part of the 76.3-to-38.8 gap is memorization; none is reported."],"forward_implications":["Document-oriented MLLMs can be given interactive grounding: users can click a region and get a grounded answer, or ask a question and see the supporting words highlighted on the page.","The data engine's cost profile (over 1,000 grounded QA pairs per dollar) makes scalable, fine-grained document data feasible without manual annotation.","The seven-task taxonomy of DOGR-Bench (grounding, referring, and combined input/output formats) gives future work a uniform way to measure and compare document grounding.","Pre-training on multi-granular parsing data also lifts text localization and recognition on standard localization benchmarks, so grounding data has a spillover benefit."],"supporting_citations":[{"why":"supplies the poster templates whose meta-annotations the re-rendering strategy corrects","marker":"[57]"},{"why":"supplies the chart images and JSON/CSV data underlying the chart parsing and instruction data","marker":"[33]"},{"why":"provides the untruncated PDF corpus used for document parsing data","marker":"[48]"},{"why":"generates the instruction-tuning questions and grounded responses in the <ocr>/<bbox> format","marker":"[16]"},{"why":"yields reading-order-aware layout detection blocks that the merge strategy uses for PDFs","marker":"[52]"},{"why":"provides comprehensive text extraction and coordinate lookup used to attach bounding boxes","marker":"[11]"},{"why":"the DocStruct4M pre-training data and DocLocal4K evaluations that anchor the baseline comparisons","marker":"[14]"}],"fun_headline_variants":["Document model points to text 38 points better than prior AI","Data engine builds model that points to text in docs with 38-point gap","Grounding gap: 76.3 vs 38.8 in document text pointing benchmark","DOGR: multimodal LLM that reads and points to text in charts, posters, PDFs"],"cache_read_input_tokens":24064,"weakest_assumption_plain":"DOGR-Bench test samples are independent of DOGR's training data, even though both are built from the same document corpus and the same automated question-generation process; no train/test overlap analysis is reported.","fun_headline_variants_meta":{"raw":{"variants":["Document model points to text 38 points better than prior AI","Data engine builds model that points to text in docs with 38-point gap","Grounding gap: 76.3 vs 38.8 in document text pointing benchmark","DOGR: multimodal LLM that reads and points to text in charts, posters, PDFs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2598,"prompt_tokens":912,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1599}},"tokens_in":528,"tokens_out":1686,"duration_ms":11030,"temperature":1.0,"reasoning_tokens":1599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:29:13.002409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply a deduplication check that compares every DOGR-Bench test image (or a re-rendered variant) and every test question-answer pair against DOGR's 703K instruction samples and 2.1M parsing samples; if a substantial fraction of test documents or grounded QA pairs appear verbatim or near-verbatim in the fine-tuning data, the headline grounding gap would be explained by memorization rather than by the data engine's claimed capability gain.","supporting_citations":[{"cited_title":"Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning","cited_arxiv_id":null,"evidence_quote":"supplies the chart images and JSON/CSV data underlying the chart parsing and instruction data"},{"cited_title":"Cc-main-2021-31-pdf- untruncated","cited_arxiv_id":null,"evidence_quote":"provides the untruncated PDF corpus used for document parsing data"},{"cited_title":"Mineru: An open-source solution for precise document content extrac- tion, 2024","cited_arxiv_id":null,"evidence_quote":"yields reading-order-aware layout detection blocks that the merge strategy uses for PDFs"},{"cited_title":"Pymupdf: Python bindings for mupdf","cited_arxiv_id":null,"evidence_quote":"provides comprehensive text extraction and coordinate lookup used to attach bounding boxes"}],"review_version":1}