{"id":"85491ae3-4b57-4b00-8f95-0340d3503825","arxiv_id":"2412.18327","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new task and synthetic dataset for recognizing human annotations in text-heavy images, with an OCR-fused Pix2Struct model that reports state-of-the-art results on the new benchmark.","lead":"This paper introduces a new visual question answering task: recognizing and extracting the text that a person has annotated in a text-heavy image. It builds a synthetic dataset, HAUR-5, and an OCR-augmented model, OCR-Mix, which reports higher accuracy than several existing vision-language models on this dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OCR-Mix's benchmark lead is confounded: it receives OCR text at inference while fine-tuned baselines do not; an OCR-augmented Pix2Struct baseline is needed.","rationale":"The reader's weakest_assumption focuses on the synthetic-to-real gap, which the paper itself concedes in the conclusion. That is a real limitation but not the most decisive weakness for the paper's central empirical claim. The reader's rationale, however, does note that 'most baselines are not fine-tuned,' which is part of the same evaluation-protocol concern I identify. My concern is more specific: OCR-Mix receives the OCR transcript at inference while the fine-tuned Pix2Struct baseline does not, so the main controlled comparison is confounded. This is directly testable with a single additional baseline. The paper still contributes a new task and dataset, so the appropriate verdict remains CONDITIONAL, matching the reader's verdict; the condition should include running this OCR-augmented baseline and releasing the data and code.","tokens_in":9298,"tokens_out":3942,"duration_ms":38573,"concrete_test":"Fine-tune Pix2Struct-base under the identical training protocol but with the same OCR transcript provided as an additional input, e.g., prepended to the question or as a separate token sequence, and evaluate on the HAUR-5 test split. If this OCR-augmented Pix2Struct baseline reaches ACC within a small margin (say 2 percentage points) of OCR-Mix's 86.47, then the claimed advantage is attributable to OCR text access rather than to the fusion module. Report per-annotation-type ACC/ANLS and results over at least three random seeds to assess variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central in-benchmark claim, that OCR-Mix outperforms other models on HAUR-5, is not yet established as a fair architecture comparison because OCR-Mix is given an asymmetric input. At inference, OCR-Mix receives the full OCR transcript of the image as an additional text input, while the only fine-tuned baseline, Pix2Struct, sees only the image and question. The task is to emit the annotated text, and that text appears verbatim in the OCR transcript; the model's job is thus reduced to selecting the annotated span, using the image to locate the annotation. The paper's own ablations (Table III and the fusion-layer study) show that adding the text modality materially changes performance, but no baseline is given the same OCR text. Therefore, the reported 14-point ACC gain over fine-tuned Pix2Struct could be entirely due to access to the OCR transcript, not to the proposed cross-attention fusion. The comparison with GPT-4o, Qwen2.5, and other web models is also unbalanced because those models are evaluated zero-shot or few-shot rather than fine-tuned, so a large gap is expected. This is more load-bearing than the synthetic-to-real gap, since it affects the core benchmark result even within the test distribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a new VQA subtask, Human Annotation Understanding and Recognition (HAUR), in which a model must output the text that a human has annotated in a text-heavy image. The authors introduce HAUR-5, a synthetic dataset of 37,702 images built from 16 public-domain novels, with five annotation styles: highlight, underline, squiggly underline, rectangular box, and paragraph marking. They also present OCR-Mix, a model that combines Pix2Struct image features with OCR-derived text features through stacked cross-attention, and they report average ACC of 86.47 and ANLS of 0.98 on the HAUR-5 test split, outperforming the evaluated baselines. The paper includes ablations of the text encoder, the number of fusion layers, and the cross-attention configuration, plus a brief qualitative 'in-wild' evaluation.","tokens_in":9509,"tokens_out":6608,"duration_ms":58785,"significance":"If the results are reproducible and fairly compared, the paper addresses a genuine and under-studied problem—locating and transcribing human annotations on documents—with a simple, well-described architecture and a publicly planned dataset. The task definition is clear, the dataset construction is documented in detail, and the authors include a candid limitation statement about synthetic-to-real transfer. However, the quantitative evidence as presented has a potentially decisive comparison asymmetry and several metric inconsistencies, so the core claim of state-of-the-art performance on HAUR is not yet established. The contribution of the dataset itself may still be valuable independent of the model result.","major_comments":[{"comment":"The central comparison is confounded by an input asymmetry. OCR-Mix is given the full OCR transcript of the image as an additional text input, while Pix2Struct—the only fine-tuned baseline—receives only the image and the question. Because the target annotation text appears verbatim in the OCR transcript, a model that copies the relevant span could solve the task without the proposed image-text fusion. The paper's ablations (Table III and Figure 4) show that the text modality matters, but they do not isolate the contribution of cross-attention fusion from the mere availability of the OCR text. Please add a baseline that receives the same OCR text input (e.g., a fine-tuned Pix2Struct or T5-style model that takes the transcript as input) before claiming that OCR-Mix's fusion module is responsible for the gains. In addition, the comparison to GPT-4o, Qwen2.5, and other web models is unbalanced because those models are evaluated zero-shot or few-shot rather than fine-tuned on HAUR-5; the reported 1.28x–4x ACC improvement is therefore not an architecture-level comparison.","section":"§IV-B2, §III-A, Table I"},{"comment":"The ANLS numbers in Table I are internally inconsistent with the stated metric. With ACC defined as exact match and ANLS computed from normalized Levenshtein distance with threshold τ=0.5, a row with ACC=0.00 cannot have an ANLS of 0.71 (e.g., LLaVA-7B on Highlight), because every prediction is wrong and the maximum possible score per sample under the stated formula is 0.5. Similarly, GLM-4's Highlight row (ACC=54.89, ANLS=0.95) is impossible: even if all incorrect answers had NL exactly 0.5, the ANLS would be at most 0.55 + 0.45×0.5 = 0.775. Please define NL precisely, provide the evaluation code or a worked example, and report exact numbers; the current table cannot be reproduced or trusted as written.","section":"§IV-B1, Eqs. (7)-(8), Table I"},{"comment":"The main results come from a single training run with the best validation model selected, and no error bars, confidence intervals, or significance tests are reported. Given the wide variation in task difficulty (e.g., Paragraph Marking ACC 61.23 vs Rect ACC 99.61 for OCR-Mix), the point-estimate comparisons in Table I may not be robust. At minimum, the authors should report the size of the test split, the number of runs, and the variance across runs, and they should test whether the observed gaps are statistically significant.","section":"§IV-C, Tables I-III"},{"comment":"The claimed generalization to real-world data is not supported by the evidence presented. The in-the-wild evaluation consists of a handful of qualitative examples (Figure 5 and Appendix D), with no quantitative measure of accuracy or agreement, and the conclusion concedes that 'the performance of the model may be different when applied to real-world data due to the differences between simulated conditions and real conditions.' To substantiate the statement that the model 'shows strong generalization capabilities in real-world scenarios' (end of §IV-E), the authors should provide a quantitative evaluation on a sample of human-annotated real images, including annotation-style labels and inter-rater or error analysis.","section":"§IV-E and Conclusion"}],"minor_comments":[{"comment":"There is a stray space before the period in 'released soon .'","section":"Abstract"},{"comment":"The sentence 'the five common types of human annotation styles present we chosen' contains a grammatical error; rephrase.","section":"§I"},{"comment":"The statement that most VQA models 'have no reference value' is contradicted by their inclusion in Table I; clarify the selection criterion.","section":"§IV-B2"},{"comment":"Define how the max over j is computed when the number of ground-truth answers M varies per question, and state the exact definition of normalized Levenshtein distance.","section":"Eq. (7)"},{"comment":"The labels 'Image-Text-Text' and 'Text-Image-Image' should be defined in terms of Eq. (3) to avoid ambiguity about which features initialize Q.","section":"§IV-D3 and Table III"},{"comment":"'In-wild' should be 'in-the-wild,' and the missing commas in 'Figures 11 12 13 14' should be fixed.","section":"Appendix D"},{"comment":"The paper says the dataset and model 'will be released soon' but provides no link or access mechanism; for reproducibility, a URL or detailed release plan should be included.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of a VQA/document-understanding venue. The main concerns are fixable through additional experiments and a corrected metric and analysis section. The ANLS inconsistencies in Table I are particularly important and should be addressed explicitly, as they affect the credibility of the headline quantitative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, the task and dataset are genuinely new: no prior dataset targets human annotations on text-heavy images, and HAUR-5 gives you five annotation styles over novel pages, about 37k images. Second, the headline result is not as clean as it looks. OCR-Mix receives the OCR transcript of the image as an extra text input, and the task is to output annotated text that is verbatim in that transcript. The only fine-tuned baseline, Pix2Struct, gets no such text. So the 14-point ACC gain over fine-tuned Pix2Struct could come from simply having the answer in the input, not from the cross-attention fusion. The paper needs a control where Pix2Struct is also given the OCR text, ideally in a way that isolates the fusion mechanism.\n\nWhat is good: The dataset construction is careful. They simulate human noise by extending highlights into adjacent sentences, and they vary the annotation geometry. They evaluate a wide range of models, including commercial ones, and report both ACC and ANLS, which is appropriate for this task. The ablation on cross-attention configuration (text-as-query vs image-as-query) is informative. The authors also explicitly concede the synthetic-to-real gap in the conclusion, which is honest.\n\nSoft spots, in order of importance. The asymmetric OCR input is the big one. Without an OCR-augmented Pix2Struct baseline, the superiority claim for OCR-Mix is unproven, even within the test distribution. Second, the synthetic-to-real generalization claim rests on a few cherry-picked in-wild examples; it is anecdotal, and the authors do not quantify it. Third, there are no error bars or significance tests; the numbers could shift with different seeds or dataset splits. Fourth, the dataset is not yet released, so the benchmark cannot be reproduced. These are fixable, but they need attention.\n\nWho is this for? Anyone working on document VQA, multimodal assistants, or OCR-centric interfaces. It is a modest but useful contribution. I would send it to review, with the expectation that the authors add the missing control and release the data. It is not a desk reject. But the central architecture claim needs to be re-validated.","headline":"Useful new task and dataset for human annotation understanding, but the model's advantage is confounded by asymmetric OCR-text input and needs a proper control.","tokens_in":10076,"tokens_out":2202,"would_cite":false,"duration_ms":20144,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that OCR-Mix, a model fusing OCR text features with Pix2Struct image features, outperforms existing VQA models and multimodal LLMs on the HAUR-5 human-annotation task.","keywords":["human annotation understanding","text-heavy images","visual question answering","OCR-Mix","HAUR-5 dataset","cross-attention fusion","multimodal learning","document image analysis"],"falsifier":"Take a set of real exam pages and annotated book excerpts with human-drawn highlights, underlines, boxes, and marginal marks, transcribe the marked text as ground truth, and run OCR-Mix against GPT-4o and a fine-tuned Pix2Struct; the central claim would be refuted if OCR-Mix's accuracy on those real images drops below its HAUR-5 levels or if the gap over GPT-4o disappears.","tokens_in":9074,"feed_emoji":"🏷️","tokens_out":8396,"duration_ms":69972,"temperature":0.7,"pith_summary":"Humans constantly annotate printed text—highlighting a sentence, underlining a key line, drawing a box around a phrase—and expect a helpful AI to know what is being pointed at. This paper introduces HAUR, a task where the model must output the exact text a human annotation marks in a text-heavy image. To make the task tractable, the authors build HAUR-5, a 37,702-image synthetic dataset covering five annotation styles, and train OCR-Mix, which combines OCR-extracted text with Pix2Struct image features through cross-attention. On this dataset, OCR-Mix reports ANLS above 0.95 on all five styles and beats GPT-4o by 1.28 to 4 times in accuracy, suggesting that a small task-specific model can supply the annotation-reading ability that general multimodal systems lack.","feed_headline":"OCR-Mix reads human annotations better than GPT-4o","feed_subtitle":"On HAUR-5, fusing OCR text with image features tops 90% accuracy on four of five annotation styles.","key_machinery":"The central object is OCR-Mix, a 1.31B-parameter encoder-decoder built on Pix2Struct's pretrained image encoder and T5-style decoder. The mechanism that carries the argument is the fusion module: OCR text extracted from the image is embedded and used as the query, while Pix2Struct image features serve as keys and values through four cross-attention layers, so the image selects which part of the OCR transcript to reproduce. A learnable embedding module encodes the OCR text, and the decoder generates the final marked-text output. This design lets a small model outperform much larger multimodal systems on the synthetic HAUR-5 distribution.","core_discovery":"On the HAUR-5 test set, OCR-Mix reaches 86.47% average accuracy and 0.98 average ANLS, with ANLS above 0.95 on every one of the five annotation styles and accuracy above 90% on highlight, underline, squiggly underline, and rectangular-box annotations (91.35%, 90.71%, 91.22%, and 99.61%, respectively). Paragraph marking is the hardest style, where accuracy falls to 61.23% but ANLS stays at 0.96. Against GPT-4o, the strongest compared model, the paper reports accuracy improvements from 1.28 to 4 times and ANLS gains of 8% to 25% depending on the annotation style. The authors interpret this as evidence that task-specific OCR-text fusion, rather than raw model scale or general instruction tuning, is what makes annotated-span extraction work.","pith_inferences":["If the approach transfers to real handwriting, the same text-query/image-key-value recipe could apply to other pointing-style VQA tasks, such as 'the word I circled' or 'the part I crossed out,' making OCR-Mix a general deictic text selector rather than a dataset-specific solution.","A testable extension would vary the amount of annotation noise—how far highlights and underlines extend into neighboring sentences—to map where OCR-Mix's advantage over GPT-4o starts to erode.","The authors' concession that simulated and real conditions differ suggests the most direct next step is a real-image benchmark; until that exists, the reported ceiling should be read as an upper bound on in-the-wild accuracy."],"forward_implications":["A model trained on HAUR-5 can serve as a preprocessing module that turns a human-annotated image into the marked plain text, which can then be fed into any language model.","The reported numbers set a quantitative benchmark for the new HAUR task, so future work can measure progress directly against OCR-Mix's 86.47% accuracy and 0.98 ANLS.","The finding that fine-tuned Pix2Struct (72.41% accuracy) outperforms all tested multimodal LLMs implies document-specialized pretraining matters more than parameter count for this task.","Using text features as the query and image features as the key/value is the configuration that works, which indicates the image's role is to select from the OCR transcript rather than the reverse."],"supporting_citations":[{"why":"Supplies the pretrained image encoder and T5-style decoder that OCR-Mix is built on.","marker":"[16]"},{"why":"Defines the ANLS evaluation metric and the 0.5 threshold used for all results.","marker":"[3]"},{"why":"Provides the aspect-ratio-preserving image preprocessing that keeps text shapes stable.","marker":"[15]"},{"why":"Frames the text-reading VQA problem that HAUR extends with human annotations.","marker":"[2]"},{"why":"Establishes document-image question answering as a target setting for structured text extraction.","marker":"[8]"},{"why":"Motivates text-heavy image understanding, the broad task family HAUR belongs to.","marker":"[1]"}],"fun_headline_variants":["OCR-Mix tops GPT-4o on annotation-heavy VQA","New model beats GPT-4o at reading human marks","OCR-Mix: 86% accuracy on human annotation VQA","Why OCR fusion beats scale for annotation VQA","HAUR-5 benchmark: OCR-Mix outperforms GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that annotations generated by automatically drawing five styles on digital document text are close enough to real human annotations that a model trained on them will keep its accuracy on actual highlighted, underlined, or hand-marked images; the paper itself warns that real-world performance may differ.","fun_headline_variants_meta":{"raw":{"variants":["OCR-Mix tops GPT-4o on annotation-heavy VQA","New model beats GPT-4o at reading human marks","OCR-Mix: 86% accuracy on human annotation VQA","Why OCR fusion beats scale for annotation VQA","HAUR-5 benchmark: OCR-Mix outperforms GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2891,"prompt_tokens":865,"completion_tokens":2026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1939}},"tokens_in":481,"tokens_out":2026,"duration_ms":13086,"temperature":1.0,"reasoning_tokens":1939,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:47:44.122053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real exam pages and annotated book excerpts with human-drawn highlights, underlines, boxes, and marginal marks, transcribe the marked text as ground truth, and run OCR-Mix against GPT-4o and a fine-tuned Pix2Struct; the central claim would be refuted if OCR-Mix's accuracy on those real images drops below its HAUR-5 levels or if the gap over GPT-4o disappears.","supporting_citations":[{"cited_title":"Pix2struct: Screenshot parsing as pretraining for visual language understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained image encoder and T5-style decoder that OCR-Mix is built on."},{"cited_title":"Scene text visual question answering,","cited_arxiv_id":null,"evidence_quote":"Defines the ANLS evaluation metric and the 0.5 threshold used for all results."},{"cited_title":"Docvqa: A dataset for vqa on document images,","cited_arxiv_id":null,"evidence_quote":"Establishes document-image question answering as a target setting for structured text extraction."},{"cited_title":"Textcaps: a dataset for image captioning with reading compre- hension,","cited_arxiv_id":null,"evidence_quote":"Motivates text-heavy image understanding, the broad task family HAUR belongs to."}],"review_version":1}