Pith. sign in

REVIEW 4 major objections 7 minor 21 references

HAUR: Human Annotation Understanding and Recognition Through Text-Heavy Images

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read The paper shows that OCR-Mix, a model fusing OCR text features with Pix2Struct image features, outperforms existing VQA models and multimodal LLMs on the HAUR-5 human-annotation task.

desk verdict Useful new task and dataset for human annotation understanding, but the model's advantage is confounded by asymmetric OCR-text input and needs a proper control. read the letter →

arxiv 2412.18327 v1 pith:NY53ZN45 submitted 2024-12-24 cs.CV

classification cs.CV
keywords humanannotationunderstandingtext-heavyimagesvisualquestionansweringOCR-MixHAUR-5datasetcross-attentionfusionmultimodallearningdocumentimageanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Humans constantly annotate printed text—highlighting a sentence, underlining a key line, drawing a box around a phrase—and expect a helpful AI to know what is being pointed at. This paper introduces HAUR, a task where the model must output the exact text a human annotation marks in a text-heavy image. To make the task tractable, the authors build HAUR-5, a 37,702-image synthetic dataset covering five annotation styles, and train OCR-Mix, which combines OCR-extracted text with Pix2Struct image features through cross-attention. On this dataset, OCR-Mix reports ANLS above 0.95 on all five styles and beats GPT-4o by 1.28 to 4 times in accuracy, suggesting that a small task-specific model can supply the annotation-reading ability that general multimodal systems lack.

What carries the argument

The central object is OCR-Mix, a 1.31B-parameter encoder-decoder built on Pix2Struct's pretrained image encoder and T5-style decoder. The mechanism that carries the argument is the fusion module: OCR text extracted from the image is embedded and used as the query, while Pix2Struct image features serve as keys and values through four cross-attention layers, so the image selects which part of the OCR transcript to reproduce. A learnable embedding module encodes the OCR text, and the decoder generates the final marked-text output. This design lets a small model outperform much larger multimodal systems on the synthetic HAUR-5 distribution.

What would settle it

Take a set of real exam pages and annotated book excerpts with human-drawn highlights, underlines, boxes, and marginal marks, transcribe the marked text as ground truth, and run OCR-Mix against GPT-4o and a fine-tuned Pix2Struct; the central claim would be refuted if OCR-Mix's accuracy on those real images drops below its HAUR-5 levels or if the gap over GPT-4o disappears.

Watch

Extended reading notes

Core claim

On the HAUR-5 test set, OCR-Mix reaches 86.47% average accuracy and 0.98 average ANLS, with ANLS above 0.95 on every one of the five annotation styles and accuracy above 90% on highlight, underline, squiggly underline, and rectangular-box annotations (91.35%, 90.71%, 91.22%, and 99.61%, respectively). Paragraph marking is the hardest style, where accuracy falls to 61.23% but ANLS stays at 0.96. Against GPT-4o, the strongest compared model, the paper reports accuracy improvements from 1.28 to 4 times and ANLS gains of 8% to 25% depending on the annotation style. The authors interpret this as evidence that task-specific OCR-text fusion, rather than raw model scale or general instruction tuning, is what makes annotated-span extraction work.

Load-bearing premise

The load-bearing premise is that annotations generated by automatically drawing five styles on digital document text are close enough to real human annotations that a model trained on them will keep its accuracy on actual highlighted, underlined, or hand-marked images; the paper itself warns that real-world performance may differ.

Editorial extensions

If this is right

  • A model trained on HAUR-5 can serve as a preprocessing module that turns a human-annotated image into the marked plain text, which can then be fed into any language model.
  • The reported numbers set a quantitative benchmark for the new HAUR task, so future work can measure progress directly against OCR-Mix's 86.47% accuracy and 0.98 ANLS.
  • The finding that fine-tuned Pix2Struct (72.41% accuracy) outperforms all tested multimodal LLMs implies document-specialized pretraining matters more than parameter count for this task.
  • Using text features as the query and image features as the key/value is the configuration that works, which indicates the image's role is to select from the OCR transcript rather than the reverse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approach transfers to real handwriting, the same text-query/image-key-value recipe could apply to other pointing-style VQA tasks, such as 'the word I circled' or 'the part I crossed out,' making OCR-Mix a general deictic text selector rather than a dataset-specific solution.
  • A testable extension would vary the amount of annotation noise—how far highlights and underlines extend into neighboring sentences—to map where OCR-Mix's advantage over GPT-4o starts to erode.
  • The authors' concession that simulated and real conditions differ suggests the most direct next step is a real-image benchmark; until that exists, the reported ceiling should be read as an upper bound on in-the-wild accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes a new VQA subtask, Human Annotation Understanding and Recognition (HAUR), in which a model must output the text that a human has annotated in a text-heavy image. The authors introduce HAUR-5, a synthetic dataset of 37,702 images built from 16 public-domain novels, with five annotation styles: highlight, underline, squiggly underline, rectangular box, and paragraph marking. They also present OCR-Mix, a model that combines Pix2Struct image features with OCR-derived text features through stacked cross-attention, and they report average ACC of 86.47 and ANLS of 0.98 on the HAUR-5 test split, outperforming the evaluated baselines. The paper includes ablations of the text encoder, the number of fusion layers, and the cross-attention configuration, plus a brief qualitative 'in-wild' evaluation.

Significance. If the results are reproducible and fairly compared, the paper addresses a genuine and under-studied problem—locating and transcribing human annotations on documents—with a simple, well-described architecture and a publicly planned dataset. The task definition is clear, the dataset construction is documented in detail, and the authors include a candid limitation statement about synthetic-to-real transfer. However, the quantitative evidence as presented has a potentially decisive comparison asymmetry and several metric inconsistencies, so the core claim of state-of-the-art performance on HAUR is not yet established. The contribution of the dataset itself may still be valuable independent of the model result.

major comments (4)
  1. [§IV-B2, §III-A, Table I] The central comparison is confounded by an input asymmetry. OCR-Mix is given the full OCR transcript of the image as an additional text input, while Pix2Struct—the only fine-tuned baseline—receives only the image and the question. Because the target annotation text appears verbatim in the OCR transcript, a model that copies the relevant span could solve the task without the proposed image-text fusion. The paper's ablations (Table III and Figure 4) show that the text modality matters, but they do not isolate the contribution of cross-attention fusion from the mere availability of the OCR text. Please add a baseline that receives the same OCR text input (e.g., a fine-tuned Pix2Struct or T5-style model that takes the transcript as input) before claiming that OCR-Mix's fusion module is responsible for the gains. In addition, the comparison to GPT-4o, Qwen2.5, and other web models is unbalanced because those models are evaluated zero-shot or few-shot rather than fine-tuned on HAUR-5; the reported 1.28x–4x ACC improvement is therefore not an architecture-level comparison.
  2. [§IV-B1, Eqs. (7)-(8), Table I] The ANLS numbers in Table I are internally inconsistent with the stated metric. With ACC defined as exact match and ANLS computed from normalized Levenshtein distance with threshold τ=0.5, a row with ACC=0.00 cannot have an ANLS of 0.71 (e.g., LLaVA-7B on Highlight), because every prediction is wrong and the maximum possible score per sample under the stated formula is 0.5. Similarly, GLM-4's Highlight row (ACC=54.89, ANLS=0.95) is impossible: even if all incorrect answers had NL exactly 0.5, the ANLS would be at most 0.55 + 0.45×0.5 = 0.775. Please define NL precisely, provide the evaluation code or a worked example, and report exact numbers; the current table cannot be reproduced or trusted as written.
  3. [§IV-C, Tables I-III] The main results come from a single training run with the best validation model selected, and no error bars, confidence intervals, or significance tests are reported. Given the wide variation in task difficulty (e.g., Paragraph Marking ACC 61.23 vs Rect ACC 99.61 for OCR-Mix), the point-estimate comparisons in Table I may not be robust. At minimum, the authors should report the size of the test split, the number of runs, and the variance across runs, and they should test whether the observed gaps are statistically significant.
  4. [§IV-E and Conclusion] The claimed generalization to real-world data is not supported by the evidence presented. The in-the-wild evaluation consists of a handful of qualitative examples (Figure 5 and Appendix D), with no quantitative measure of accuracy or agreement, and the conclusion concedes that 'the performance of the model may be different when applied to real-world data due to the differences between simulated conditions and real conditions.' To substantiate the statement that the model 'shows strong generalization capabilities in real-world scenarios' (end of §IV-E), the authors should provide a quantitative evaluation on a sample of human-annotated real images, including annotation-style labels and inter-rater or error analysis.
minor comments (7)
  1. [Abstract] There is a stray space before the period in 'released soon .'
  2. [§I] The sentence 'the five common types of human annotation styles present we chosen' contains a grammatical error; rephrase.
  3. [§IV-B2] The statement that most VQA models 'have no reference value' is contradicted by their inclusion in Table I; clarify the selection criterion.
  4. [Eq. (7)] Define how the max over j is computed when the number of ground-truth answers M varies per question, and state the exact definition of normalized Levenshtein distance.
  5. [§IV-D3 and Table III] The labels 'Image-Text-Text' and 'Text-Image-Image' should be defined in terms of Eq. (3) to avoid ambiguity about which features initialize Q.
  6. [Appendix D] 'In-wild' should be 'in-the-wild,' and the missing commas in 'Figures 11 12 13 14' should be fixed.
  7. [Availability] The paper says the dataset and model 'will be released soon' but provides no link or access mechanism; for reproducibility, a URL or detailed release plan should be included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the OCR-input asymmetry is a baseline-fairness concern, not a circular derivation.

full rationale

The paper's central claim is an empirical benchmark result on HAUR-5, a synthetic dataset the authors constructed. The model is trained on an 80% split and evaluated on a held-out 10% test split, so the reported ACC/ANLS numbers are not by construction equal to the training objective or to the dataset-generation labels. The fact that HAUR-5 is synthetic and author-generated affects external validity and generalization claims, not circularity; the paper itself concedes this in the conclusion, noting that 'the performance of the model may be different when applied to real-world data due to the differences between simulated conditions and real conditions.' The OCR transcript given to OCR-Mix at inference does contain the answer as a substring, and this creates an asymmetric comparison with fine-tuned Pix2Struct, which does not receive that transcript. However, this is a control/fairness issue: the model's task is to select the annotated span from the transcript using the image as the localization signal, and the evaluation is computed against ground-truth labels derived from the annotation-generation process, not from the model's own predictions or from a fitted parameter renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled via citation. The in-the-wild demonstrations are anecdotal and weak evidence for generalization, but an anecdotal example is not a circular step. Therefore no pattern in the enumerated circularity taxonomy applies, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central claims rest on the HAUR-5 dataset and the OCR-Mix architecture. The dataset is the main new artifact, but it is synthetic and unaudited by external evaluators, so the assumptions about realism and transfer are load-bearing. The free parameters are hyperparameters and dataset-construction choices, not derived quantities.

free parameters (4)
  • Number of fusion layers N = 4
    Selected by validation experiments in Section IV-D2; the model performs best at N=4.
  • Initial learning rate = 1e-5
    Set in implementation details without a reported sweep.
  • Maximum text chunk length = 500 tokens
    Dataset construction choice that shapes text-length distribution and annotation contexts.
  • Annotation noise simulation rule = random extension to adjacent sentence
    Hand-chosen simulation of human imprecision for highlight, underline, and squiggly underline annotations.
assumptions (3)
  • domain assumption Synthetic annotations approximate real human annotations.
    Dataset generation uses random sentence selection and artificial noise; there is no quantitative validation against real annotated data, and the conclusion admits performance may differ on real data.
  • domain assumption Pix2Struct pretraining transfers to annotation recognition.
    The model uses Pix2Struct's image encoder and decoder as the base, assuming its document and webpage pretraining helps with the HAUR task.
  • domain assumption OCR text extraction captures all relevant text in the image.
    The text features are derived from OCR output; if OCR misses or garbles text, the model cannot use it. This is not explicitly verified on HAUR-5.
invented entities (1)
  • HAUR-5 dataset
    purpose: Benchmark and training set for the HAUR task, generated synthetically from Project Gutenberg novels.
    The dataset is self-constructed and not yet released; no external benchmark confirms it represents typical human annotations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HAUR: Human Annotation Understanding and Recognition Through Text-Heavy Images." pith.science (2026). https://pith.science/paper/NY53ZN45

@misc{pith2026241218327,
  author       = {Pith},
  title        = {Pith review of: HAUR: Human Annotation Understanding and Recognition Through Text-Heavy Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NY53ZN45}},
  note         = {Machine review of arXiv:2412.18327}
}
read the original abstract

Vision Question Answering (VQA) tasks use images to convey critical information to answer text-based questions, which is one of the most common forms of question answering in real-world scenarios. Numerous vision-text models exist today and have performed well on certain VQA tasks. However, these models exhibit significant limitations in understanding human annotations on text-heavy images. To address this, we propose the Human Annotation Understanding and Recognition (HAUR) task. As part of this effort, we introduce the Human Annotation Understanding and Recognition-5 (HAUR-5) dataset, which encompasses five common types of human annotations. Additionally, we developed and trained our model, OCR-Mix. Through comprehensive cross-model comparisons, our results demonstrate that OCR-Mix outperforms other models in this task. Our dataset and model will be released soon .

Figures

Figures reproduced from arXiv: 2412.18327 by the authors.

Figure 1
Figure 1. A real-life scenario to explain the motivation behind our task. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our OCR-Mix Model. • We identified an unresolved issue in human annota￾tion recognition in real-world scenarios and propose the HAUR task. This task aims to accurately interpret human annotations and can also assist large language models, enabling them to answer related questions. • We propose the HAUR-5 dataset, comprising 37,702 images with five common types of human annotations. We believe tha… view at source ↗
Figure 3
Figure 3. A simple image showcasing the contents of our HAUR-5 dataset: The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: For N = 0, we concatenated the text and image features and input them into the decoder via a learnable FC layer. The results indicate that the model performs best when N = 4. As N increases further, performance declines. We believe that for small values of N, increasin…
Figure 5
Figure 5. Figure 5: Experiment with human-labeled images in real-world scenarios: Red [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: The PM method is slightly more complex; we first [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 6
Figure 6. Figure 6: An example of human annotation in real life. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Full Text Length Distribution [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Ground Truth Length Distribution C. Query and Prompt As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 11
Figure 11. Figure 11: A example of in-wild data results [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 10
Figure 10. Figure 10: Query in the VQA Model and Prompt in the VLLM [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 12
Figure 12. Figure 12: A example of in-wild data results [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: A example of in-wild data results [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

  1. [1]

    Textcaps: a dataset for image captioning with reading compre- hension,

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh, “Textcaps: a dataset for image captioning with reading compre- hension,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 . Springer, 2020, pp. 742–758

  2. [2]

    Towards vqa models that can read,

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach, “Towards vqa models that can read,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 8317–8326

  3. [3]

    Scene text visual question answering,

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas, “Scene text visual question answering,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 4291–4301

  4. [4]

    Recipeqa: A challenge dataset for multimodal comprehension of cook- ing recipes,

    Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis, “Recipeqa: A challenge dataset for multimodal comprehension of cook- ing recipes,” arXiv preprint arXiv:1809.00812 , 2018

  5. [5]

    Visualmrc: Machine reading comprehension on document images,

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida, “Visualmrc: Machine reading comprehension on document images,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2021, vol. 35, pp. 13878– 13888

  6. [6]

    Textbook question answering under instructor guidance with memory networks,

    Juzheng Li, Hang Su, Jun Zhu, Siyu Wang, and Bo Zhang, “Textbook question answering under instructor guidance with memory networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3655–3663

  7. [7]

    Viocrvqa: Novel benchmark dataset and vision reader for visual question answering by understanding vietnamese text in images,

    Huy Quang Pham, Thang Kien-Bao Nguyen, Quan Van Nguyen, Dan Quang Tran, Nghia Hieu Nguyen, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen, “Viocrvqa: Novel benchmark dataset and vision reader for visual question answering by understanding vietnamese text in images,” arXiv preprint arXiv:2404.18397 , 2024

  8. [8]

    Docvqa: A dataset for vqa on document images,

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar, “Docvqa: A dataset for vqa on document images,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2021, pp. 2200– 2209

Show all 21 references
  1. [9]

    Infographicvqa,

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar, “Infographicvqa,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 1697–1706

  2. [10]

    Iterative answer prediction with pointer-augmented multimodal trans- formers for textvqa,

    Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach, “Iterative answer prediction with pointer-augmented multimodal trans- formers for textvqa,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 9992–10002

  3. [11]

    Tap: Text- aware pre-training for text-vqa and text-caption,

    Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo, “Tap: Text- aware pre-training for text-vqa and text-caption,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. ...

  4. [12]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    Wonjae Kim, Bokyung Son, and Ildoo Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International conference on machine learning . PMLR, 2021, pp. 5583–5594

  5. [13]

    Pali: A jointly-scaled multilingual language-image model,

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al., “Pali: A jointly-scaled multilingual language-image model,” arXiv preprint arXiv:2209.06794 , 2022

  6. [14]

    Pali-3 vision language models: Smaller, faster, stronger,

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdul- mohsin, Piotr Padlewski, et al., “Pali-3 vision language models: Smaller, faster, stronger,” arXiv preprint arXiv:2310.09199 , 2023

  7. [15]

    Screenai: A vision-language model for ui and infographics understanding,

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor C ˘arbune, Jason Lin, Jindong Chen, and Abhanshu Sharma, “Screenai: A vision-language model for ui and infographics understanding,” arXiv preprint arXiv:2402.04615 , 2024

  8. [16]

    Pix2struct: Screenshot parsing as pretraining for visual language understanding,

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova, “Pix2struct: Screenshot parsing as pretraining for visual language understanding,” in International Conferen...

  9. [17]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024

  10. [18]

    Improved baselines with visual instruction tuning,

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, “Improved baselines with visual instruction tuning,” 2023

  11. [19]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966 , 2023

  12. [20]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution,

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin, “Qwen2-vl: Enhancing vision- language model’s ...

  13. [500]

    This limitation was based on our understanding of real- world scenarios: even if the entire text on a page is long, people typically do not photograph the entire page but instead focus on capturing key content and its relevant context, as shown in the Figure 6. Fig. 6. An exam...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.