REVIEW 3 major objections 8 minor 19 references
InstructOCR: Instruction Boosting Scene Text Spotting
T0 review · 3 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that feeding scene text spotters short human-language instructions, such as "recognize text that starts with A," improves spotting accuracy and transfers to VQA.
desk verdict Plausible instruction-conditioning idea, but the key ablation is confounded by extra training epochs, so the central causal claim is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is instruction-conditioned cross-modal decoding. The model adds a trainable BERT text encoder to the SPTS-style sequence-generation spotter; instruction tokens are encoded and fused with ResNet-50 visual features via cross-attention before the autoregressive decoder predicts per-instance sequences $[x, y, t]$. The instructions themselves come from ten templates over text attributes—length thresholds, specific words, first/last characters, numeric vs alphabetic content—so they are generated programmatically from existing word-level annotations at no extra labeling cost. At inference, the fixed instruction "<Recognize all text>" makes the model behave as a standard spotter, while variable instructions filter which instances to output, and in VQA the question is encoded by the same text encoder.
What would settle it
Retrain the SPTS-V2 baseline with the exact pretraining recipe used for InstructOCR-dagger—including TextOCR, HierText, and the same instruction-epoch schedule—but without any instructions. If that baseline's lexicon-free Hmean on Total-Text and ICDAR2015 matches or exceeds InstructOCR-dagger, then the reported instruction gains are explained by data and training budget rather than by the instruction mechanism. Alternatively, run an ablation of InstructOCR with the text encoder removed and the same data; if it matches, the effect is not linguistic.
Extended reading notes
Core claim
InstructOCR is an end-to-end encoder-decoder scene text spotter that takes both an image and a text instruction, and emits a token sequence of center points and transcriptions for the text instances that match the instruction. The paper's central discovery is that adding this linguistic conditioning—a BERT text encoder whose features are cross-attended with ResNet-50 image features—improves reading accuracy even when the instruction at test time is simply "recognize all text." The paper reports consistent gains over the SPTS line of single-point spotters, with the largest lexicon-free improvements on Total-Text and ICDAR2015, and shows that instruction-based pretraining on 0.2M scene text images improves downstream scene-text VQA by 2.1 points on ST-VQA and 2.6 points on TextVQA with a 78M-parameter model. The authors present this as evidence that aligning human language instructions with visual text helps OCR-related tasks, and that the gains extend beyond spotting into question answering.
Load-bearing premise
The headline results assume that comparing InstructOCR trained with extra TextOCR and HierText data against baselines trained on their original data isolates the benefit of instructions, even though the paper does not retrain those baselines with the extra data.
Editorial extensions
If this is right
- The same architecture can serve as a controllable spotter: one model answers "recognize all text," "recognize text of five characters," or "recognize text starting with A" without retraining.
- Instruction pretraining on text-spotting data carries over to VQA, improving TextVQA by 2.6 and ST-VQA by 2.1 over identical OCR-only pretraining.
- Because instructions are derived from existing annotations, the method adds no data-collection cost and can be applied to any word-level scene text dataset.
- The gains are achieved by a 78M-parameter model with a ResNet-50 image encoder and BERT text encoder, in contrast to the billion-parameter vision-language models used on the same VQA benchmarks.
- Adding larger spotting corpora (TextOCR, HierText) to the pretraining mixture further lifts results, suggesting the mechanism scales with data.
Reading between the lines
- Our inference: the cleanest way to separate the instruction effect from the data effect is to retrain the SPTS baselines on the same combined pretraining set; the paper does not do this, so the dagger-row gains should be read as an upper bound on the instruction benefit.
- Our inference: instruction diversity can be viewed as a freeform data augmentation over the output space; the same template idea could be applied to document parsing or key-information extraction, where attributes like font, color, or position could be encoded as instructions.
- Our inference: since the VQA gain appears after instruction pretraining, instruction-following during pretraining may act as a general text-understanding regularizer, which could transfer to other OCR-heavy multimodal tasks beyond VQA.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes InstructOCR, an encoder-decoder scene text spotter built on the SPTS single-point sequence-generation paradigm. It adds a BERT text encoder that ingests instructions (ten hand-designed templates over text attributes such as length, start/end characters, and content), fuses text and image features via cross-attention, and is trained with a fixed '<Recognize all text>' instruction during the final 50-epoch stage. On Total-Text, ICDAR2015, and ICDAR2013, InstructOCR is reported to outperform SPTS-v1/v2, and with additional TextOCR/HierText pretraining it claims state-of-the-art results. The same framework is applied to ST-VQA and TextVQA, where instruction-based pretraining gives +2.1 and +2.6 points over pretraining without instructions.
Significance. The central idea—that conditioning a small sequence-generation text spotter on natural-language instructions improves text understanding and transfers to VQA—is interesting and plausible. If the effect is real, it offers a cheap way to boost performance without extra supervision. The paper releases code, ablates the text encoder separately from the instruction mechanism, and tests on four public benchmarks. However, the significance is currently limited by two confounds: the SOTA comparison is not data-matched, and the instruction ablation is not epoch-matched. These issues make the headline quantitative claims unverifiable from the reported experiments, though the qualitative idea remains promising.
major comments (3)
- [Comparison with Scene Text Spotting Methods, Table 2] In Table 2, the InstructOCR† row adds TextOCR and HierText to the training data while the comparison methods (e.g., DNTextSpotter at 84.5 on Total-Text None) are not given the same data. The abstract and Section 4 claim state-of-the-art results on this basis, but with matched data InstructOCR is actually below DNTextSpotter on Total-Text None (83.4 versus 84.5). This is a data-quantity confound rather than evidence for the instruction mechanism. Please either re-run the strongest baselines with the same additional data, or restrict the SOTA claim to a clear statement that it holds only with extra pretraining data.
- [Implementation Details and Tables 4–5] The instruction gains in Tables 4 and 5 are potentially an epoch effect. The Implementation Details state that after 200+40 epochs, 'instructions are added, and the model is further trained for another 50 epochs.' The no-instruction rows in Table 5 (row 2, 'W' without 'INS') and Table 4 (the '✓ ✓ ✘' row) are not described as receiving the same additional 50-epoch stage with the fixed '<Recognize all text>' instruction. Consequently, the reported +2.1 ST-VQA and +2.6 TextVQA improvements, and the +0.4% in Table 4, may be due to simply training longer. Please provide a no-instruction baseline that trains for exactly the same total number of epochs (including the final 50) with the fixed instruction, or otherwise show that the gains persist when total training iterations are held constant.
- [Ablation Studies, Tables 4 and 5] All results are single runs without error bars or multiple seeds. The smallest reported effect is +0.4% in Table 4, which is within typical seed-to-seed variability for end-to-end scene text spotting models of this size. Even for the larger +2.1/+2.6 effects in Table 5, no variance estimate is given. To support the causal claim that instructions help, please report mean and standard deviation over at least three independent runs, or provide training curves showing the instruction and no-instruction runs converge to different plateaus.
minor comments (8)
- [Discussion] The text says improvements 'without a dictionary' on ICDAR2013 and ICDAR2015, but Table 2 has no 'None' column for those datasets; the cited numbers correspond to the 'G' (generic lexicon) column. Please clarify the lexicon setting or add the missing None columns.
- [Table 4 caption] The symbols 'W', 'T', and 'INS' are not defined in the caption itself; the surrounding text defines them but a self-contained caption would avoid confusion.
- [Figure 1] Figure 1 contains the typo 'Insturction' in both example labels; please correct.
- [Loss Function, Eq. (1)] Equation (1) is written as a maximization over a weighted sum, but the text describes standard cross-entropy minimization; please rewrite the loss in the usual negative-log-likelihood form with clear notation for the target sequence.
- [Applicability to Scene-Text VQA] The VQA section does not describe how the question is encoded or how the answer sequence is produced in the Method; Figure 3 shows the design but the text should state that the question is passed through the BERT text encoder and the answer is a token sequence.
- [Introduction, related work] The paper states that 'to the best of our knowledge, this is the first work that innovatively integrates human language instructions into scene text spotting,' but STEP (cited) already uses regular expressions as a form of textual control; please qualify the novelty claim to distinguish natural-language instructions from regular-expression control.
- [Instructions Generation, Table 1] The instruction templates in Table 1 include both 'Recognition of all text in the image' and the inference instruction '<Recognize all text>'; please make the wording consistent.
- [Related Work] The Related Work section has a heading 'None Sequence-based Method' that should read 'Non-sequence-based Method'.
Circularity Check
No significant circularity: the instruction-boosting claim is evaluated on held-out benchmarks and does not reduce to its training inputs or to a self-citation chain.
full rationale
The paper's central claims are empirical comparisons on held-out test sets (Total-Text, ICDAR2013/2015, ST-VQA, TextVQA), so no reported number is defined from a fitted parameter or from the evaluation target. The instruction mechanism is a conditional sequence-generation objective (Eq. 1) trained with instructions sampled from text attributes; at inference the instruction is fixed to '<Recognize all text>', and gains are measured against ablated baselines. This is a standard supervised generalization setup rather than a self-definitional reduction. The only self-citation, ODM (Duan, Fu et al. 2024), appears in the introduction as motivation for vision-text pretraining and is not used to derive InstructOCR's architecture or results, so it is not load-bearing under the stated rules. The possible training-schedule confound (the 'instructions are added ... trained for another 50 epochs' versus the no-instruction rows in Tables 4 and 5) and the extra-data comparison for the InstructOCR-dagger row are substantive experimental-design concerns, but neither makes a prediction equivalent to an input by construction. Accordingly the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Scene text spotting can be represented as autoregressive prediction of discretized center points and transcription tokens.
- domain assumption Pretrained BERT embeddings capture the semantics of short human-language instructions well enough to guide visual decoding.
- ad hoc to paper The ten hand-designed instruction templates span the text attributes that matter for scene text spotting.
- standard math Cross-entropy token loss with equal token weights is an appropriate training objective for the mixed coordinate and transcription sequence.
Cite this review
Pith. "Pith review of InstructOCR: Instruction Boosting Scene Text Spotting." pith.science (2026). https://pith.science/paper/NKGOP63C
@misc{pith2026241215523,
author = {Pith},
title = {Pith review of: InstructOCR: Instruction Boosting Scene Text Spotting},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKGOP63C}},
note = {Machine review of arXiv:2412.15523}
}
read the original abstract
In the field of scene text spotting, previous OCR methods primarily relied on image encoders and pre-trained text information, but they often overlooked the advantages of incorporating human language instructions. To address this gap, we propose InstructOCR, an innovative instruction-based scene text spotting model that leverages human language instructions to enhance the understanding of text within images. Our framework employs both text and image encoders during training and inference, along with instructions meticulously designed based on text attributes. This approach enables the model to interpret text more accurately and flexibly. Extensive experiments demonstrate the effectiveness of our model and we achieve state-of-the-art results on widely used benchmarks. Furthermore, the proposed framework can be seamlessly applied to scene text VQA tasks. By leveraging instruction strategies during pre-training, the performance on downstream VQA tasks can be significantly improved, with a 2.6% increase on the TextVQA dataset and a 2.1% increase on the ST-VQA dataset. These experimental results provide insights into the benefits of incorporating human language instructions for OCR-related tasks.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[6]
arXiv preprint arXiv:2307.12270
Context Perception Paral- lel Decoder for Scene Text Recognition. arXiv preprint arXiv:2307.12270. Duan, C.; Fu, P.; et al
-
[7]
arXiv preprint arXiv:2309.03895
Instructdiffu- sion: A generalist modeling interface for vision tasks. arXiv preprint arXiv:2309.03895. He, K.; Zhang, X.; et al
-
[10]
arXiv preprint arXiv:2111.15664, 7:
Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7:
-
[11]
arXiv preprint arXiv:2304.02643
Segment anything. arXiv preprint arXiv:2304.02643. Kittenplon, Y .; Lavi, I.; et al
-
[12]
arXiv preprint arXiv:2303.05499
Grounding dino: Marrying dino with grounded pre-training for open-set object detec- tion. arXiv preprint arXiv:2303.05499. Liu, X.; et al
-
[13]
SPTS v2: Single-Point Scene Text Spotting
SPTS v2: single-point scene text spotting. arXiv preprint arXiv:2301.01635. Long, S.; Qin, S.; et al
-
[14]
arXiv preprint arXiv:2309.11419
Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419. Lyu, P.; Liao, M.; et al
-
[15]
Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt. In 2017 14th IAPR international con- ference on document analysis and recognition (ICDAR), vol- ume 1, 1454–1459. IEEE. Peng, D.; Wang, X.; et al
work page 2017
Show all 19 references
-
[16]
arXiv preprint arXiv:2312.02694
Upocr: Towards unified pixel-level ocr interface. arXiv preprint arXiv:2312.02694. Qiao, L.; Chen, Y .; et al
-
[19]
arXiv preprint arXiv:2310.05126
Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126. Ye, M.; Zhang, J.; et al
-
[2013]
In 12th international conference on document analysis and recognition, 1484–1493
ICDAR 2013 robust reading competition. In 12th international conference on document analysis and recognition, 1484–1493. IEEE. Karatzas, D.; et al
2013
-
[2015]
In 13th international conference on document analysis and recognition, 1156–1160
ICDAR 2015 competition on ro- bust reading. In 13th international conference on document analysis and recognition, 1156–1160. IEEE. Kil, J.; et al
2015
-
[2017]
In 2017 14th IAPR international conference on document anal- ysis and recognition (ICDAR), volume 1, 935–942
Total-text: A compre- hensive dataset for scene text detection and recognition. In 2017 14th IAPR international conference on document anal- ysis and recognition (ICDAR), volume 1, 935–942. IEEE. Devlin, J.; Chang, M.-W.; et al
2017
-
[2018]
arXiv preprint arXiv:1810.04805
Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Dosovitskiy, A.; Beyer, L.; et al
-
[2020]
arXiv preprint arXiv:2010.11929
An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Du, Y .; Chen, Z.; et al
2010 arXiv
-
[2021]
arXiv preprint arXiv:2109.10852
Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852. Chen, T.; Saxena, S.; et al
-
[2022]
arXiv preprint arXiv:2205.14100
Git: A generative image- to-text transformer for vision and language. arXiv preprint arXiv:2205.14100. Wang, P.; Zhang, C.; et al
-
[2023]
arXiv preprint arXiv:2308.12966
Qwen-VL: A Versatile Vision- Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966. Biten, A. F.; Tito, R.; et al
-
[2024]
arXiv preprint arXiv:2403.19128
OmniParser: A Unified Frame- work for Text Spotting, Key Information Extraction and Ta- ble Recognition. arXiv preprint arXiv:2403.19128. Wang, H.; Lu, P.; et al
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.