{"id":"bcb98f24-31a6-499e-b7f3-d13ee0eef68d","arxiv_id":"2608.04424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ADOPD 2026 extends ADOPD 2024 with semantic tags, captions, and grounded reasoning traces, and its DocCount benchmark shows top VLMs reach just 72.85% exact-match accuracy.","lead":"ADOPD 2026 adds human-written captions, semantic tags, and generated reasoning traces to the ADOPD 2024 page-layout dataset, creating a benchmark for region-grounded document understanding. On its new dense counting test, DocCount, the best tested vision-language model answers only 72.85% of questions correctly, so the task remains open.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DocCount's zero-shot 'struggle' claim rests on ground-truth counts whose grouping and class-definition consistency are never measured; label ambiguity may explain part of the gap.","rationale":"I read the paper in good faith. The localization experiments are extensive, the agentic grouping ablation is carefully controlled, and the label pipeline does include a GPT-5.5 verifier plus a human missing-label review, which is genuine independent support. The DocCount numbers are probably real measurements. The load-bearing question is what those measurements mean. The paper's central unsolved-problem claim depends on DocCount being a fair, unambiguous benchmark, and the paper asserts reliability without measuring it. The reader's weakest assumption was that the generated CoT traces are faithful; that is a real dataset-quality issue, but the headline counting claim is more directly threatened by ground-truth count ambiguity, because the benchmark's exact-match metric cannot separate counting errors from disagreements about how to group or interpret regions under the stated definitions. The qualitative examples in A.13 show exactly this failure mode: several strong models make errors that are primarily definition-following or grouping decisions rather than enumeration failures. A re-annotation test would settle whether the low accuracies are stable against label ambiguity. This does not overturn the paper or the conditional verdict; if the test passes, the central claim is considerably stronger, and if it fails, the paper needs ambiguity-aware metrics, refined definitions, or an adjudicated subset. I therefore keep the reader's CONDITIONAL verdict unchanged while flagging this as the item to verify before broader claims are accepted.","tokens_in":36573,"tokens_out":6505,"duration_ms":79830,"concrete_test":"Re-annotate all DocCount pages, or a stratified sample of at least 100 pages per target class, with two or more independent annotators who see only the original image, the class definition, and the evaluation prompt (no polygon overlays, no ground-truth instance lists). Compute per-class and overall exact-count agreement, plus a chance-corrected index such as Cohen's kappa. If exact-match agreement is below roughly 85% or kappa below 0.7, recompute the Table 5 accuracies on the adjudicated subset to see how much of the VLM gap is attributable to label-policy ambiguity rather than counting ability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that current VLMs cannot reliably perform dense semantic counting on documents (Table 5, 37.56–72.85% exact accuracy). For that conclusion to hold, DocCount's ground-truth counts must be both correct and unambiguous under the released class definitions. This is the least secure link in the paper. Construction (Sec. 4.3.1) filters to a top-1 class with more than 3 objects and uses a GPT-5.5 verifier plus a manual pass to check whether selected polygons are mislabeled or missing, but it never measures inter-annotator agreement on the classification and grouping decisions that determine each count. The evaluation prompt contains the same definitions, and several of those definitions leave grouping unresolved. In the paper's own qualitative examples, 'Photograph' does not specify whether spatially contiguous product shots on a shared background count separately (GT=4, Kimi/Qwen=6, Claude=4); 'Brand Logo' does not settle aircraft livery and fuselage marks against designed marks (GT=7, GPT-5.5=11); 'Chart/Graph' does not settle whether a legend is an independent chart (GT=5, Kimi=6). With roughly 442 examples and no per-class sizes or confidence intervals, the ranking between 72.85% and 70.81% is also fragile. Without reliability evidence, the headline that the task 'remains far from solved' may partly reflect an uncalibrated label policy rather than a grounded-counting deficit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ADOPD 2026, an enrichment of the ADOPD 2024 corpus with human-cleaned captions, entity-level semantic tags, and generated chain-of-thought (CoT) traces tied to polygon anchors, and it formalizes a 'visual anchor' abstraction. It evaluates three capabilities: Doc2Box/Doc2Mask localization with fine-tuned detectors and an agentic grouping pipeline; 12-class region tagging with zero-shot and fine-tuned VLMs; and DocCount, a dense counting benchmark on 400+ document pages on which 13 VLMs score between 37.56% and 72.85% exact accuracy. The paper's central claims are that document decomposition exhibits a genuine domain gap, that semantic tagging is long-tail difficult, and that dense semantic counting on documents remains unsolved.","tokens_in":36913,"tokens_out":8387,"duration_ms":70632,"significance":"The resource contribution is potentially substantial: 120k pages, roughly 2.5M tagged elements, a released project page and code, and a carefully constructed train/validation/test split. The localization fine-tuning experiments are internally consistent and convincingly demonstrate a real domain gap. The DocCount zero-shot results, if the ground-truth counts are reliable, would provide a useful diagnostic for VLM counting in documents. However, the benchmark's label reliability and the factual grounding of the CoT traces are not established, so the significance of the headline counting claim is currently conditional on additional validation.","major_comments":[{"comment":"The central claim that dense semantic counting is unsolved rests on DocCount ground-truth counts, but the paper never measures the reliability of the classification and grouping decisions that determine each count. Section 4.3.1 filters to the top-1 class, uses a GPT-5.5 verifier and manual missing-label review, yet no inter-annotator agreement is reported and the class definitions leave unresolved cases. In the A.13 examples, spatially contiguous product shots on a shared background are scored as 4 by the ground truth, 6 by Kimi/Qwen, and 4 by Claude for Photograph; the Chart/Graph example shows the legend panel changing the answer between 5 and 6 across models. With 442 samples and no per-class sample sizes or confidence intervals, the ranking between 72.85% and 70.81% is fragile, and the claim that the task 'remains far from solved' may partly reflect label-policy ambiguity. Please report a double-annotated subset with agreement on grouping decisions, per-class counts and confidence intervals, and a sensitivity analysis of model rankings under alternative grouping policies.","section":"Sec. 4.3.1-4.3.2, Table 5"},{"comment":"The Anchor-CoT traces are a core new resource, but their factual grounding is not verified. The text states that only polygon labels are checked by a VLM plus human review; the generator is 'prompted to write natural-language reasoning' and the reasoning text itself is not checked for correctness. Consequently, the dataset's claim to teach 'anchor-grounded reasoning' is not supported: traces could hallucinate region descriptions while the polygon list remains correct. Add a verification step for the reasoning statements, or explicitly scope the contribution to geometrically controlled traces with unverified prose.","section":"Sec. 3.2, A.12"},{"comment":"The tagging evaluation is based on n=360 (30 regions per class) with no error bars or significance tests. Differences such as Qwen2.5-VL single-image vs dual-image (35.8 vs 36.4) or Gemma4-12B single vs dual (43.9 vs 48.6) are plausibly within noise, and per-class estimates from 30 samples are unstable. The claim that fine-tuning 'is helpful in recovering the rare classes' needs confidence intervals or bootstrap replicates, and the per-class sample sizes should be stated explicitly.","section":"Sec. 4.2, Table 4"},{"comment":"Table 5 reports per-class accuracies but not the per-class sample sizes within DocCount, so the overall accuracy is hard to interpret. If, for example, Photograph samples greatly outnumber Brand Logo samples, the aggregate 72.85% would be dominated by one class. The paper should report the number of documents per class in DocCount and discuss how class imbalance affects the aggregate ranking and the conclusion that the task remains unsolved.","section":"Table 5 and Sec. 4.3.2"}],"minor_comments":[{"comment":"The sentence ending 'definition-following, semantic understanding, and dense grounding on documents.n' contains a stray 'n'; the final sentence of the analysis paragraph also appears truncated and should be completed.","section":"Sec. 4.3.2"},{"comment":"The number of DocCount samples is given as 442 in Table 2 but as '400+' in the Table 5 caption and in Section 4.3.2; please unify the reported sample count.","section":"Table 2, Table 5"},{"comment":"The text says 13 state-of-the-art VLMs are evaluated, but Table 5 appears to list 12 model rows; please check the model count and the table rendering.","section":"Sec. 3.2 and Table 5"},{"comment":"The mF1 metric is defined differently for LocateAnything (confidence-free F1@IoU Mean) and for non-VLM detectors (confidence-swept F1 at IoU 0.5); the paper notes this, but the table caption should repeat the caveat so the numbers are not read as directly comparable across columns.","section":"Sec. 4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is DocCount label reliability; if the authors supply the reliability analysis, per-class counts, and CoT verification, the paper would be acceptable. The self-citation pattern is not disqualifying, and the localization results are solid. I recommend major revision rather than rejection because the core resource is potentially valuable and the identified issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper is worth referee time. It builds a real extension of ADOPD 2024—semantic tags, cleaned captions, CoT traces over polygon anchors—and the DocCount benchmark is a useful diagnostic: 13 VLMs land between 37.56% and 72.85% exact count, which does suggest current models cannot reliably do dense document counting. The localization experiments are also solid, with a plausible domain-gap story and an honest account of the agentic grouping and its geometric guard.\n\nThe soft spot is the one the stress-test flags. DocCount's ground-truth counts depend on class definitions that leave grouping and boundary decisions underdetermined, and the paper never measures inter-annotator reliability on those decisions. The paper's own examples illustrate the problem: six product shots on a shared background are tagged as 4 Photograph regions; a legend in the corner counts as a separate chart for some models but not in the GT; aircraft livery vs. designed marks flips Brand Logo counts. With n=442, no per-class sample sizes, and no confidence intervals, the exact leaderboard—72.85% vs 70.81%—is likely within noise. That does not kill the central claim that the task is unsolved, but it does mean the headline gap is partly a label-policy gap, and the authors should quantify it.\n\nTwo smaller things. The generated Anchor-CoT traces are asserted to be 'grounded' but the reasoning text is never verified for factual correctness; the polygon labels are checked, not the prose. And despite the 'Thinking-with-Anchors' framing, no model is ever trained end-to-end on the traces; the paper presents components (localization, tagging, counting, agentic grouping) rather than the pipeline the title implies. Both are fixable in revision—verify a sample of traces, and either implement the pipeline or tone down the framing.\n\nBottom line: send it to review. The dataset and benchmark are genuinely useful resources, the counting gap is real even if its exact size is uncertain, and the paper is honest about many of its own limitations. For your own work, the DocCount benchmark is citable; I'd want reliability numbers before leaning on the precise ranking.","headline":"A genuinely useful dataset extension and counting benchmark with a real zero-shot gap, but the headline numbers depend on label-policy ambiguity that is never measured; still worth peer review.","tokens_in":37484,"tokens_out":2799,"would_cite":true,"duration_ms":38531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 120k-page dataset argues that document understanding should be judged as reasoning over visual anchors, and shows that today's vision-language models still fail at dense document counting, with the best scoring 72.85%.","keywords":["document understanding","visual anchors","grounded reasoning","chain-of-thought","dense counting","semantic tagging","layout analysis","vision-language models"],"falsifier":"Mask the numbered polygon placeholders in a random sample of Anchor-CoT traces and ask independent annotators to say which page region each reasoning sentence refers to; if the prose alone does not recover the counted instances or yields a different count, the claimed grounding is not real.","tokens_in":36389,"feed_emoji":"🧮","tokens_out":6391,"duration_ms":67045,"temperature":0.7,"pith_summary":"ADOPD 2026 is an attempt to turn document page decomposition into a testbed for grounded reasoning. The paper starts from a 120k-page corpus of dense OCR boxes and human-drawn entity polygons, then adds human-cleaned captions, closed-vocabulary semantic tags, and generated chain-of-thought traces in which each counted region is tied to numbered polygon anchors. Its central claim is that boxes, masks, labels, and text blocks should be treated as one shared vocabulary of visual anchors, and that this representation lets a model point to evidence while answering, not just localize. The paper's benchmark evidence is DocCount, 442 dense counting questions over four region classes, where 13 current vision-language models score between 37.56% and 72.85% exact accuracy. The conclusion a sympathetic reader should take is that dense semantic counting on documents is not solved, and that evaluating region semantics separately from geometry exposes long-tail failures that layout benchmarks hide.","feed_headline":"Best VLM scores 72.85% on dense document counting","feed_subtitle":"A 120k-page dataset ties every count to verified polygon anchors, exposing where vision-language models still fail.","key_machinery":"The load-bearing object is the visual anchor: a text block, visual entity, semantic label, bounding box, or polygon mask treated as one shared vocabulary for detection, tagging, and reasoning. On top of it, the construction procedure adds human-cleaned captions, a 30-class tag taxonomy (normalized to 12 classes for evaluation), and Anchor-CoT traces generated by a VLM from simplified polygon masks, where numbered placeholders like [[1]] and [[2]] keep the prose geometrically controlled while the count and instance order remain deterministic functions of verified labels. DocCount is the resulting 442-sample benchmark over four target classes—brand logo, photograph, table, and chart/graph—scored by exact integer accuracy with ten question phrasings. This machinery lets the paper separate perception errors from policy errors: a model must follow the class definition, see the whole page, and enumerate the right instances.","core_discovery":"The core discovery, stated on the paper's own terms, is that document understanding should be organized around visual anchors—document regions that carry geometry, semantic role, and relations to other regions—and that current vision-language models handle this poorly. The dataset builds three capabilities on that abstraction: region-level semantic tagging, unified generation of text and visual entities with coordinates or polygons, and anchor-grounded chain-of-thought reasoning. Evaluations show a genuine domain gap: off-the-shelf detectors and segmenters are near failure on documents, while fine-tuning lifts AP by 50–70 points; zero-shot VLMs overuse common tags and miss the long tail; and on DocCount the best of 13 models reaches only 72.85%, with the task explicitly described as far from solved. The paper therefore argues that document benchmarks should move beyond localization to verify whether a model can use a region as evidence for an answer.","pith_inferences":["If Anchor-CoT traces are faithful, fine-tuning on them should transfer to counting accuracy; a direct test would compare a VLM trained on the grounded traces against one trained on answer-only labels on DocCount.","The same anchor vocabulary could be turned into a broader evidence-verification benchmark, for example asking a model to justify a document answer by citing specific polygons and having a deterministic check score the citation.","The over-splitting result suggests that detection AP may understate usable performance; a system that merges proposals by reasoning could outperform a better detector at the same task."],"forward_implications":["If DocCount is a fair benchmark, any claim of general document reasoning ability should be tested on it, because current best accuracy of 72.85% leaves a quarter of pages wrong.","Fine-tuning a grounder on ADOPD 2026 recovers long-tail tags such as background image and color block, whereas zero-shot models almost never emit them, so the dataset provides a way to measure semantic tagging separately from localization.","The agentic grouping workflow, which reasons over numbered boxes and applies a geometric guard, closes 49% of the merge-oracle gap on text-box decomposition without retraining, showing that semantic grouping can repair detector over-fragmentation.","Unified grounding outputs—boxes and polygons emitted as structured anchors—can be reused and checked by downstream reasoning systems, making answers verifiable rather than free-form."],"supporting_citations":[{"why":"The predecessor dataset that supplies the inherited OCR boxes and entity polygons for the 120k pages.","marker":"[22]"},{"why":"The visual-primitives paradigm that motivates constructing counting questions from numbered region primitives and the Anchor-CoT format.","marker":"[39]"},{"why":"The generative grounder used as the unified detection, segmentation, and tagging baseline in the experiments.","marker":"[65]"},{"why":"The visual-counting benchmark that motivates DocCount by showing that fluent multimodal responses do not guarantee reliable counting.","marker":"[16]"},{"why":"The real-time detector whose over-recall text-box pool is refined by the agentic grouping workflow.","marker":"[54]"}],"fun_headline_variants":["Document counting: best model reaches only 72.85% accuracy","Anchor-based doc reasoning exposes counting gaps in VLMs","VLMs max out at 72.85% on dense document counting","Reasoning with anchors: new benchmark for document intelligence","ADOPD 2026: grounding doc reasoning with visual anchors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reasoning text in the Anchor-CoT traces is generated by a VLM without verification of factual grounding—only polygon labels are checked—so the central promise that the dataset teaches anchor-grounded reasoning depends on those traces not hallucinating.","fun_headline_variants_meta":{"raw":{"variants":["Document counting: best model reaches only 72.85% accuracy","Anchor-based doc reasoning exposes counting gaps in VLMs","VLMs max out at 72.85% on dense document counting","Reasoning with anchors: new benchmark for document intelligence","ADOPD 2026: grounding doc reasoning with visual anchors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2873,"prompt_tokens":997,"completion_tokens":1876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1790}},"tokens_in":613,"tokens_out":1876,"duration_ms":12178,"temperature":1.0,"reasoning_tokens":1790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:02.264771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Mask the numbered polygon placeholders in a random sample of Anchor-CoT traces and ask independent annotators to say which page region each reasoning sentence refers to; if the prose alone does not recover the counted instances or yields a different count, the claimed grounding is not real.","supporting_citations":[],"review_version":1}