{"id":"236102c9-6559-4560-b6c5-3c724cc96e1b","arxiv_id":"2505.05446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Converting document images into markup-language representations before answering questions improves visual document understanding, and the released DocMark datasets enable an adaptive markup generation pipeline.","lead":"This paper trains visual language models to convert document images into structured markup languages such as Markdown, HTML, JSON, and TikZ, then uses that markup as intermediate context when answering questions. It introduces two large training sets, DocMark-Pile and DocMark-Instruct, and reports gains on several visual document understanding benchmarks with 2B and 8B models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over SOTA are confounded by unmatched fine-tuning: InternVL2 baselines are off-the-shelf, while DocMark adds 3.8M + 624k training samples, so the adaptive markup mechanism is not isolated.","rationale":"The paper proposes a plausible and interesting pipeline, and the internal ablations (Table 5 and Figure 5) suggest that the CoT component and DocMark-Pile both help. Credit is due for a large constructed dataset and for some controlled comparisons, even if they use different base models. The reader's weakest assumption focuses on the trustworthiness of ChatGPT-3.5-generated CoT labels; that is a real concern because no human validation or error rate is reported, and the prompt includes the gold answer, which may produce post-hoc rationalizations rather than genuine extraction. However, the more load-bearing issue for the central claim is experimental attribution: without an equivalently fine-tuned InternVL2 baseline, the 'substantially outperform SOTA' claim conflates the method with additional training data. This is directly testable and would settle whether the method matters. The 8B deficits on DocVQA and InfoVQA further undercut the broad wording of the claim, though the paper's 'nearly all' hedges partially cover this. Conditional acceptance is appropriate: the idea is publishable, but the headline comparison needs a matched-baseline experiment and the CoT labels need validation before the claim is accepted at face value.","tokens_in":19173,"tokens_out":5340,"duration_ms":64772,"concrete_test":"Fine-tune the released InternVL2-2B and InternVL2-8B checkpoints on exactly the same DocMark-Pile and DocMark-Instruct QA data as DocMark, but keep only the second-round answer (Q2/A2) and remove the two-round CoT context and A1 supervision. Evaluate both the full two-round model and this non-CoT matched control on all benchmarks in Table 2, with at least three seeds and reported standard errors. If the non-CoT control closes the gap to within noise, the headline gains are data-driven; if the full two-round model still wins by the stated margins, the markup-context mechanism is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that adaptive markup generation 'substantially outperforms existing state-of-the-art MLLMs' rests on Table 2, but the comparison is not controlled. DocMark starts from InternVL2 and then adds 3.8M DocMark-Pile pretraining samples plus 624k DocMark-Instruct fine-tuning samples, whereas the InternVL2 rows are the released checkpoints without equivalent task-specific fine-tuning. The reported margins (e.g., +8.3 AI2D, +3.4 ChartQA at 2B) could thus reflect the extra training data and compute rather than the markup-context mechanism. The supplementary's Table 5 attempts a control, but it is built on Qwen2-VL and LLaVA-OneVision, not on the InternVL2 backbones used in the main claim, and its 'DocMark(w/o CoT)' condition still differs from vanilla in both DocMark-Pile pretraining and fine-tuning data. The main text also overstates consistency: at 8B, DocMark is 1.0 point below InternVL2 on DocVQA and 4.5 points below on InfoVQA (Table 2). Until an equivalently fine-tuned baseline is evaluated, the unique contribution of the adaptive markup pipeline remains unidentified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive markup-language-generation pipeline for visual document understanding. The model is first pretrained on DocMark-Pile (approximately 3.8M image-to-markup pairs covering plain text, Markdown, LaTeX, HTML, JSON, and TikZ) and then fine-tuned on DocMark-Instruct (624k chain-of-thought-style annotations) where it is trained to first output a markup-language context snippet and then answer the question. The authors instantiate this on InternVL2-style backbones with 2B and 8B language models, evaluate on document, chart, OCR, webpage, and math benchmarks, and claim consistent improvements over existing state-of-the-art MLLMs. The supplementary includes ablations showing that removing either proposed dataset degrades performance and that CoT fine-tuning yields further gains on Qwen2-VL and LLaVA-OneVision backbones.","tokens_in":19436,"tokens_out":4209,"duration_ms":42653,"significance":"The core idea is attractive and well motivated: converting documents into structured, language-model-friendly markup forms as a grounding step could reduce hallucination and improve reasoning. The two released datasets are large, described in detail, and would be useful resources if released as promised. The internal ablations (Fig. 5 and supplementary Tab. 5) provide evidence that the proposed training data and the CoT fine-tuning contribute positively to performance. However, the headline claim that the method 'substantially outperforms existing state-of-the-art MLLMs' is not yet supported because the main comparison is uncontrolled, and the quality of the automatically generated CoT annotations is not validated. The significance of the work therefore hinges on additional controlled experiments that isolate the adaptive-markup mechanism from the effect of simply adding more task-specific training data.","major_comments":[{"comment":"The main SOTA comparison is not controlled. DocMark is fine-tuned on the training splits of TextVQA, DocVQA, InfographicsVQA, ChartQA, DVQA, XFUND, and Geo170K (Sec. 4.1) plus DocMark-Pile and DocMark-Instruct, whereas the InternVL2 and other baselines are used off-the-shelf. The reported margins (e.g., +8.3 on AI2D and +3.4 on ChartQA at 2B) could therefore reflect the additional 3.8M + 624k training samples and compute rather than the adaptive markup mechanism. The supplementary Tab. 5 does not resolve this because it uses Qwen2-VL and LLaVA-OneVision rather than the InternVL2 backbones of the main claim, and its 'DocMark(w/o CoT)' still differs from vanilla in both pretraining and fine-tuning data. Please add an equivalently fine-tuned baseline on the same backbone (e.g., InternVL2 fine-tuned on the same DocMark-Instruct data without the two-round context-generation format, or on the original QA pairs) to isolate the mechanism.","section":"Sec. 4.3, Table 2"},{"comment":"The trustworthiness of the 624k CoT labels is load-bearing but not validated. The construction first converts images to markup with the pretrained model, then asks ChatGPT-3.5 to extract the relevant context (Fig. 7 in the supplement). No human validation, error-rate estimate, or consistency check is reported, and the supplementary admits the markup drops style information such as font sizes and colors (Sec. C.1). If the generated markup is incorrect or if ChatGPT-3.5 extracts plausible but wrong context, the fine-tuning labels encode those errors, and the reported gains may partly reflect learning annotation noise rather than improved document understanding. Please provide a sampled human evaluation of context correctness and a breakdown of how often the 'unclear' fallback is used.","section":"Sec. 3.2.2 and supplementary Sec. C.1"},{"comment":"The claim that DocMark 'consistently outperforms state-of-the-art MLLMs' is overstated. At 8B, DocMark is 1.0 point below InternVL2 on DocVQA (89.8 vs. 90.8) and 4.5 points below on InfoVQA (68.3 vs. 72.8). Since no error bars or repeated-run variance are reported, the remaining differences (e.g., +1.0 on AI2D, +2.0 on ChartQA at 8B) may not be statistically significant. Please report confidence intervals or run-to-run variance, and soften the consistency claim accordingly.","section":"Sec. 4.3, Table 2"}],"minor_comments":[{"comment":"The text says 'We visualize the distribution of different language parsing tasks in Fig. 2,' but Fig. 2 is the overview of the dataset; the domain distribution appears in Fig. 3. Please correct the cross-reference.","section":"Sec. 3.1.2"},{"comment":"The metrics in Table 1 are not fully defined. For example, the text-recognition column appears as a single number for each method, but the underlying benchmark [38] reports multiple metrics; please state the exact aggregation (e.g., average accuracy over subtasks). Similarly, the DaTikZ metric is described only as 'average of image similarity and code similarity,' without specifying the similarity measures.","section":"Sec. 4.2, Table 1"},{"comment":"The 'WebQA accuracy as proposed in VisualWebBench' is ambiguous because VisualWebBench contains multiple webpage-understanding tasks. Please specify the exact task and split used.","section":"Sec. 4.3"},{"comment":"The special-token notation is inconsistent: the two-round template uses '<type>' and '</type>', while the dataset descriptions use concrete tokens such as '<md>' and '<json>'. Please clarify that '<type>' is a placeholder for one of the concrete markup tokens.","section":"Sec. 3.2.1"},{"comment":"Please specify the exact version of InternVL2 used (e.g., InternVL2-2B vs. InternVL2-2B-448px) and confirm that the dynamic-resolution settings (maximum sub-images, image size) match those used in the DocMark training; otherwise the comparison to the InternVL2 rows is even harder to interpret.","section":"Sec. 4.1"},{"comment":"The 'In-house data' rows in DocMark-Instruct have no description of how the images and questions were collected or filtered. Please add a short description and any curation criteria.","section":"Supplementary Tab. 3"}],"recommendation":"major_revision","confidential_remarks":"This paper makes a large dataset contribution and ships code and model releases, which I view positively. The main risk to the paper's central claim is the uncontrolled comparison in Table 2; I would like to see an equivalently fine-tuned baseline on the same backbone before the 'substantial improvement' claim is accepted. The label-quality issue for the ChatGPT-generated CoT data is also important for reproducibility and for interpreting the ablation gains. These are addressable in a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the adaptive markup idea is real and the datasets are a contribution. The model choosing among plain text, Markdown, LaTeX, HTML, JSON, and TikZ as an intermediate context step is a genuine combination that I don't think prior single-format converters (Nougat, WebSight, OneChart, DeTikZify) cover. At scale, 3.8M pretraining pairs plus 624k CoT annotations is serious engineering, and the ablations (Fig 5, Tab 5) do show the added data helping. Code and models are promised, which helps reproducibility.\n\nThe problem is that the paper's main evidence doesn't test the mechanism. Table 2 compares DocMark, fine-tuned on the training splits of several evaluated benchmarks, against InternVL2 off-the-shelf. The margins could simply be the 3.8M + 624k extra samples and compute. The text overstates: at 8B DocMark is 1.0 below InternVL2 on DocVQA and 4.5 below on InfoVQA, yet the paper claims consistent SOTA. The supplementary's Table 5 tries to control this, but on Qwen2-VL and LLaVA-OneVision backbones, not InternVL2, and its \"w/o CoT\" condition still includes DocMark-Pile pretraining, so it doesn't isolate the adaptive markup/CoT component. This is the load-bearing soft spot, and it is addressable: train InternVL2 on the same data volume with a direct-answer target, or at least on DocMark-Pile without the CoT round, and rerun.\n\nSecond soft spot: the DocMark-Instruct labels are bootstrap-generated. Their own pretrained model produces markup, then ChatGPT-3.5 extracts context; no human validation, error rate, or consistency check is reported, and Sec C.1 admits style information is dropped. That isn't fatal, because the final evaluation is on public test sets, but it means the fine-tuning targets may encode annotation noise, and some gains could be learning that noise. A small human-checked sample would firm this up.\n\nMinor: no error bars anywhere, and a few table numbers have formatting ambiguity. The internal ablations do support that DocMark-Pile and DocMark-Instruct help over the base model, so the pipeline itself is not a mirage.\n\nWho should read it: people working on document VQA or structured intermediate representations for MLLMs. It deserves a serious referee; the right outcome is probably major revision, not rejection, assuming the authors add a matched baseline and a validation sample for the CoT labels.","headline":"A useful data recipe and a plausible mechanism, but the headline SOTA claim is confounded by unmatched fine-tuning; the markup-as-CoT idea deserves a careful revision, not a pass.","tokens_in":20015,"tokens_out":3146,"would_cite":true,"duration_ms":34047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generating the document's markup first, then answering from it, lets 2B and 8B models outperform larger systems.","keywords":["visual document understanding","multimodal large language model","markup language generation","chain-of-thought reasoning","structured document representation","instruction tuning","context grounding","document parsing"],"falsifier":"Take a random sample of DocMark-Instruct, have human annotators verify that each extracted context is correct and sufficient to answer the question, and measure the error rate; if a substantial fraction of labels are wrong, or if retraining on a human-verified subset shrinks the reported gains, the central claim would be undermined.","tokens_in":18954,"feed_emoji":"📄","tokens_out":9687,"duration_ms":80525,"temperature":0.7,"pith_summary":"The paper's central claim is that a multimodal large language model can be trained to first convert a document image into the most suitable markup language (Markdown, JSON, HTML, TikZ, LaTeX, or plain text), then extract the contextually relevant portion of that markup to answer a question, and this two-step, chain-of-thought-style procedure substantially improves visual document understanding. To test this, the authors construct two datasets: DocMark-Pile, 3.8M pretraining pairs covering six markup languages, and DocMark-Instruct, 624k instruction annotations with the model's own markup-derived context as intermediate rationales. On a broad set of benchmarks, their 2B and 8B models are reported to surpass much larger document-specific multimodal systems. If true, the result suggests that structured code-like intermediate representations can ground MLLM answers and reduce hallucination in text-rich images.","feed_headline":"Markup-first training helps small multimodal models beat larger ones","feed_subtitle":"By first emitting the document's markup as a reasoning step, 2B and 8B models surpass larger specialized systems.","key_machinery":"The load-bearing mechanism is the adaptive markup generation pipeline, implemented as a two-round conversational template: (Q1) 'To answer the question, extract the relevant context from the image' and (A1) a markup-language snippet delimited by special type tokens; then (Q2) 'Based on the image and extracted context, answer the question' and (A2) the final answer. The first training stage gives the model the ability to parse diverse documents into six markup formats (plain text, Markdown, LaTeX, HTML, JSON, TikZ). The second stage fine-tunes on DocMark-Instruct, whose annotations pair each question with an extracted context, so the model learns to produce relevant, condensed markup before answering. The context tokens are a small fraction of the total token budget compared with image tokens, so the extra reasoning step adds little computational overhead.","core_discovery":"The discovery is that letting the model choose and emit the markup language itself, rather than prescribing one, gives it an intermediate reasoning step that disentangles perception from reasoning. In the two-round dialogue, the model first answers 'extract the relevant context from the image' with a markup snippet delimited by type tokens, and then answers the original question using both the image and that snippet. The paper argues that this mimics how a person scans a document, locates the relevant region, and then reads off the answer. The ablation studies attribute most of the improvement to this chain-of-thought component, not merely to the extra pretraining data.","pith_inferences":["A natural extension is to apply the same adaptive markup idea to other structured output domains, such as SVG or spreadsheet formulas, or to structured inputs like audio waveforms and 3D scenes.","The performance ceiling is partly set by the third-party LLM used to build labels; retraining with human-verified contexts or with a stronger extractor may push accuracy further or reveal what the current labels get wrong.","One testable implication is that gains should be largest on questions requiring precise layout or numeric lookup, and smallest on questions answerable from general knowledge; a per-category breakdown would test this.","The method could also act as a data flywheel: model-generated markup that provably supports a correct answer could be re-used as training data for future document parsing models."],"forward_implications":["If the central claim holds, small open models (2B, 8B) can achieve document understanding accuracy that previously required much larger or domain-specific systems.","The pipeline's intermediate markup output can be inspected, giving a human-readable rationale for a model's answer and enabling targeted error diagnosis.","Because the context tokens are cheap compared with image tokens, the method can be added to existing MLLMs with modest inference cost.","The ablation indicates that the chain-of-thought context-generation component, not just extra training data, is what drives the improvements."],"supporting_citations":[{"why":"Supplies the base vision-language architecture that DocMark is built on and modified with two-stage training.","marker":"[9]"},{"why":"Supplies the dynamic-resolution image partitioning that lets the model handle high-resolution, complex document images.","marker":"[8]"},{"why":"Provides the synthetic webpage screenshots and matching HTML code used to construct the HTML parsing task in DocMark-Pile.","marker":"[27]"},{"why":"Provides the TikZ source files and recompilable examples used for the scientific-diagram parsing task and its evaluation.","marker":"[2]"},{"why":"Provides the table images with annotations that are converted to Markdown for the table parsing portion of DocMark-Pile.","marker":"[75]"},{"why":"Serves as both a core evaluation benchmark and a source of document QA examples with Markdown context in DocMark-Instruct.","marker":"[44]"},{"why":"Serves as both a core evaluation benchmark and a source of chart QA examples with JSON context in DocMark-Instruct.","marker":"[42]"},{"why":"Serves as both an evaluation benchmark and a source of scene-text QA examples used in DocMark-Instruct.","marker":"[55]"}],"fun_headline_variants":["Adaptive markup lets small models outthink larger ones in visual docs","Markup-first reasoning lets 2B and 8B models top the charts","Choose-your-own-markup strategy makes small multimodal models SOTA","Letting the model pick its markup language unlocks visual doc reasoning","Intermediate markup step lifts small multimodal models past bigger ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DocMark-Instruct labels are generated without human validation: the authors' own pretrained model converts images to markup, and a general-purpose chat LLM extracts the answer-relevant context, so if either step produces plausible but wrong output, the fine-tuning labels encode that error and the reported gains would partly reflect learning annotation noise.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive markup lets small models outthink larger ones in visual docs","Markup-first reasoning lets 2B and 8B models top the charts","Choose-your-own-markup strategy makes small multimodal models SOTA","Letting the model pick its markup language unlocks visual doc reasoning","Intermediate markup step lifts small multimodal models past bigger ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3027,"prompt_tokens":872,"completion_tokens":2155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":2065}},"tokens_in":488,"tokens_out":2155,"duration_ms":13669,"temperature":1.0,"reasoning_tokens":2065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:03:53.376385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of DocMark-Instruct, have human annotators verify that each extracted context is correct and sufficient to answer the question, and measure the error rate; if a substantial fraction of labels are wrong, or if retraining on a human-verified subset shrinks the reported gains, the central claim would be undermined.","supporting_citations":[{"cited_title":"Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the base vision-language architecture that DocMark is built on and modified with two-stage training."},{"cited_title":"DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ","cited_arxiv_id":null,"evidence_quote":"Provides the TikZ source files and recompilable examples used for the scientific-diagram parsing task and its evaluation."},{"cited_title":"Docvqa: A dataset for vqa on document images","cited_arxiv_id":null,"evidence_quote":"Serves as both a core evaluation benchmark and a source of document QA examples with Markdown context in DocMark-Instruct."},{"cited_title":"ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning","cited_arxiv_id":null,"evidence_quote":"Serves as both a core evaluation benchmark and a source of chart QA examples with JSON context in DocMark-Instruct."},{"cited_title":"Towards vqa models that can read","cited_arxiv_id":null,"evidence_quote":"Serves as both an evaluation benchmark and a source of scene-text QA examples used in DocMark-Instruct."}],"review_version":1}