{"id":"59bdfb8e-cf3a-4711-8832-e4472272bcb2","arxiv_id":"2412.15523","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"InstructOCR conditions an autoregressive scene text spotter on human-language instruction templates and reports gains on Total-Text, ICDAR2015, ICDAR2013, TextVQA, and ST-VQA.","lead":"InstructOCR adds human-language instructions to a scene text spotting model, so the model can be told things like 'recognize text starting with A' or 'recognize only numbers'. On standard benchmarks the authors report gains over prior spotters, and using instruction pretraining also improves two scene-text VQA datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instruction gains in Tables 4–5 may be an epoch effect: 'instructions are added' for 50 extra epochs, and the no-instruction rows are not shown to receive equal training.","rationale":"The reader's weakest assumption correctly identifies the InstructOCR† SOTA row as confounded by additional TextOCR and HierText data. My stress-test focuses on an earlier and more central link in the argument: the controlled-looking comparisons that are supposed to demonstrate the instruction mechanism itself. The Implementation Details explicitly insert an additional 50-epoch stage when instructions are introduced, and the paper does not state that the no-instruction controls receive the same number of epochs. This makes the headline causal claim vulnerable to a train-for-longer/better-schedule explanation. I still consider conditional acceptance appropriate, because the underlying idea is plausible, the code is released, and the proposed matched-epoch rerun is simple to execute. The verdict label therefore remains CONDITIONAL/UNCHANGED, but the acceptance conditions should include the matched-training-control experiment rather than only the SOTA comparison fix.","tokens_in":12278,"tokens_out":6830,"duration_ms":59069,"concrete_test":"Rerun Table 5 with matched total pretraining epochs: row 2 should receive the same 50 extra epochs as row 3, but with the fixed instruction '<Recognize all text>' instead of the ten attribute templates, and likewise for Table 4's no-instruction row. Report per-seed mean and standard deviation over at least three seeds for ST-VQA ANLS and TextVQA accuracy. If the W-vs-W+INS gap closes to within noise, the paper's central claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core claim that human-language instructions improve scene text spotting and transfer to VQA rests on Tables 4 and 5. In Implementation Details, the paper trains for 200 epochs, then 40 more at higher resolution, and then states: 'Then, instructions are added, and the model is further trained for another 50 epochs.' Table 5's row 2 ('W') is described only as pretraining on 0.2M OCR data without instructions; row 3 adds 'INS'. Nothing in the text says row 2 received the same 50 additional epochs with the fixed '<Recognize all text>' instruction. If row 2 was trained for only 200+40 epochs, the reported +2.1 ST-VQA / +2.6 TextVQA gains could be due to additional training iterations rather than to instruction conditioning. The same ambiguity affects Table 4: the final 'INS' row may be the only row that includes the extra 50-epoch stage. This confound is more load-bearing than the data confound on the InstructOCR† row, because it undermines the causal conclusion that 'incorporating human language instructions ... enhances understanding' rather than merely the SOTA comparison. The paper also reports no error bars or multi-seed runs, so the small effects (+0.4% in Table 4, +2.1/+2.6 in Table 5) cannot be separated from schedule or seed variation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InstructOCR, an encoder-decoder scene text spotter built on the SPTS single-point sequence-generation paradigm. It adds a BERT text encoder that ingests instructions (ten hand-designed templates over text attributes such as length, start/end characters, and content), fuses text and image features via cross-attention, and is trained with a fixed '<Recognize all text>' instruction during the final 50-epoch stage. On Total-Text, ICDAR2015, and ICDAR2013, InstructOCR is reported to outperform SPTS-v1/v2, and with additional TextOCR/HierText pretraining it claims state-of-the-art results. The same framework is applied to ST-VQA and TextVQA, where instruction-based pretraining gives +2.1 and +2.6 points over pretraining without instructions.","tokens_in":12580,"tokens_out":6500,"duration_ms":48624,"significance":"The central idea—that conditioning a small sequence-generation text spotter on natural-language instructions improves text understanding and transfers to VQA—is interesting and plausible. If the effect is real, it offers a cheap way to boost performance without extra supervision. The paper releases code, ablates the text encoder separately from the instruction mechanism, and tests on four public benchmarks. However, the significance is currently limited by two confounds: the SOTA comparison is not data-matched, and the instruction ablation is not epoch-matched. These issues make the headline quantitative claims unverifiable from the reported experiments, though the qualitative idea remains promising.","major_comments":[{"comment":"In Table 2, the InstructOCR† row adds TextOCR and HierText to the training data while the comparison methods (e.g., DNTextSpotter at 84.5 on Total-Text None) are not given the same data. The abstract and Section 4 claim state-of-the-art results on this basis, but with matched data InstructOCR is actually below DNTextSpotter on Total-Text None (83.4 versus 84.5). This is a data-quantity confound rather than evidence for the instruction mechanism. Please either re-run the strongest baselines with the same additional data, or restrict the SOTA claim to a clear statement that it holds only with extra pretraining data.","section":"Comparison with Scene Text Spotting Methods, Table 2"},{"comment":"The instruction gains in Tables 4 and 5 are potentially an epoch effect. The Implementation Details state that after 200+40 epochs, 'instructions are added, and the model is further trained for another 50 epochs.' The no-instruction rows in Table 5 (row 2, 'W' without 'INS') and Table 4 (the '✓ ✓ ✘' row) are not described as receiving the same additional 50-epoch stage with the fixed '<Recognize all text>' instruction. Consequently, the reported +2.1 ST-VQA and +2.6 TextVQA improvements, and the +0.4% in Table 4, may be due to simply training longer. Please provide a no-instruction baseline that trains for exactly the same total number of epochs (including the final 50) with the fixed instruction, or otherwise show that the gains persist when total training iterations are held constant.","section":"Implementation Details and Tables 4–5"},{"comment":"All results are single runs without error bars or multiple seeds. The smallest reported effect is +0.4% in Table 4, which is within typical seed-to-seed variability for end-to-end scene text spotting models of this size. Even for the larger +2.1/+2.6 effects in Table 5, no variance estimate is given. To support the causal claim that instructions help, please report mean and standard deviation over at least three independent runs, or provide training curves showing the instruction and no-instruction runs converge to different plateaus.","section":"Ablation Studies, Tables 4 and 5"}],"minor_comments":[{"comment":"The text says improvements 'without a dictionary' on ICDAR2013 and ICDAR2015, but Table 2 has no 'None' column for those datasets; the cited numbers correspond to the 'G' (generic lexicon) column. Please clarify the lexicon setting or add the missing None columns.","section":"Discussion"},{"comment":"The symbols 'W', 'T', and 'INS' are not defined in the caption itself; the surrounding text defines them but a self-contained caption would avoid confusion.","section":"Table 4 caption"},{"comment":"Figure 1 contains the typo 'Insturction' in both example labels; please correct.","section":"Figure 1"},{"comment":"Equation (1) is written as a maximization over a weighted sum, but the text describes standard cross-entropy minimization; please rewrite the loss in the usual negative-log-likelihood form with clear notation for the target sequence.","section":"Loss Function, Eq. (1)"},{"comment":"The VQA section does not describe how the question is encoded or how the answer sequence is produced in the Method; Figure 3 shows the design but the text should state that the question is passed through the BERT text encoder and the answer is a token sequence.","section":"Applicability to Scene-Text VQA"},{"comment":"The paper states that 'to the best of our knowledge, this is the first work that innovatively integrates human language instructions into scene text spotting,' but STEP (cited) already uses regular expressions as a form of textual control; please qualify the novelty claim to distinguish natural-language instructions from regular-expression control.","section":"Introduction, related work"},{"comment":"The instruction templates in Table 1 include both 'Recognition of all text in the image' and the inference instruction '<Recognize all text>'; please make the wording consistent.","section":"Instructions Generation, Table 1"},{"comment":"The Related Work section has a heading 'None Sequence-based Method' that should read 'Non-sequence-based Method'.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable conference-style paper with a plausible idea and released code. The main risk is the two confounds (data and epoch matching). The authors should be asked to provide a matched comparison before the SOTA and instruction claims can be accepted. Given the scope, I would recommend major revision rather than rejection, because the central mechanism is testable and the paper already separates the text encoder from instructions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the novelty is real, but the internal ablation that is supposed to prove instructions help has a schedule confound, and your stress-test note lands. The paper trains 200 epochs, then 40 more at higher resolution, then adds instructions for another 50 epochs. Table 4's no-instruction row and Table 5's 'W' row do not appear to receive those 50 extra epochs. So the +0.4 on ICDAR2015 and +2.1/+2.6 on VQA could be more-training-iterations effects rather than instruction-conditioning effects. That is load-bearing because the discussion and conclusion explicitly claim human-language instructions enhance understanding. No error bars or multi-seed runs make small effects even harder to interpret.\n\nWhat the paper does do well: the specific combination is genuinely new to scene text spotting. A trainable BERT text encoder, ten attribute-based instruction templates generated for free from existing annotations, and cross-attention fusion inside an autoregressive single-point spotter is not in the SPTS, TCM, FastTCM, STEP, or UNITS line. The VQA transfer is a sensible extension and the authors are honest that their 78M model is far smaller than the LMM baselines. The code link is a plus, and the writing is clear.\n\nThe SOTA claim is the second soft spot. The InstructOCR-dagger row adds TextOCR and HierText to its pretraining, but the comparison methods are left on their original training sets. That is not a fair comparison, and the paper even acknowledges UNITS uses those datasets while not giving the same treatment to others. Dropping 'state-of-the-art' from the abstract would be more honest.\n\nThis is fixable. Run the no-instruction baseline for the same 50 additional epochs with a fixed '<Recognize all text>' instruction, and either redo the SOTA comparison with matched data or soften the language. The instruction idea is worth testing, and the architecture is interesting, but the current evidence does not establish the causal claim.\n\nWho is this for? People working on sequence-based text spotting and instruction-tuned OCR will want to read it, and it deserves a serious referee. I would not cite the instruction gains yet, but I would send it back for re-ablation rather than desk-reject.","headline":"Plausible instruction-conditioning idea, but the key ablation is confounded by extra training epochs, so the central causal claim is not yet supported.","tokens_in":13105,"tokens_out":1732,"would_cite":false,"duration_ms":17432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that feeding scene text spotters short human-language instructions, such as \"recognize text that starts with A,\" improves spotting accuracy and transfers to VQA.","keywords":["scene text spotting","instruction-based OCR","BERT text encoder","sequence generation","text attributes","scene text VQA","human language instructions","single-point spotting"],"falsifier":"Retrain the SPTS-V2 baseline with the exact pretraining recipe used for InstructOCR-dagger—including TextOCR, HierText, and the same instruction-epoch schedule—but without any instructions. If that baseline's lexicon-free Hmean on Total-Text and ICDAR2015 matches or exceeds InstructOCR-dagger, then the reported instruction gains are explained by data and training budget rather than by the instruction mechanism. Alternatively, run an ablation of InstructOCR with the text encoder removed and the same data; if it matches, the effect is not linguistic.","tokens_in":12113,"feed_emoji":"🔤","tokens_out":5920,"duration_ms":35441,"temperature":0.7,"pith_summary":"Scene text spotters traditionally read an image and output every text instance; InstructOCR instead lets a short human-language instruction condition what the model detects and recognizes. The paper claims this instruction mechanism improves accuracy on standard benchmarks—gains of +2.9 on Total-Text, +0.3 on ICDAR2013, and +6.3 on ICDAR2015 in lexicon-free settings relative to SPTS-V1—and that the same pretraining transfers to scene-text VQA, adding +2.1 on ST-VQA and +2.6 on TextVQA. A trainable BERT text encoder fuses instruction semantics with visual features through cross-attention, and ten templates built from text attributes (length, start/end character, numeric vs alphabetic) generate diverse instructions for free from existing annotations. If correct, the work shows that explicit linguistic directives are a cheap, effective way to make small sequence-generation OCR models more accurate and more adaptable.","feed_headline":"Language instructions lift scene text spotting by 6.3 points","feed_subtitle":"Same instruction pretraining raises TextVQA by 2.6 points with a 78M-parameter model.","key_machinery":"The central mechanism is instruction-conditioned cross-modal decoding. The model adds a trainable BERT text encoder to the SPTS-style sequence-generation spotter; instruction tokens are encoded and fused with ResNet-50 visual features via cross-attention before the autoregressive decoder predicts per-instance sequences $[x, y, t]$. The instructions themselves come from ten templates over text attributes—length thresholds, specific words, first/last characters, numeric vs alphabetic content—so they are generated programmatically from existing word-level annotations at no extra labeling cost. At inference, the fixed instruction \"<Recognize all text>\" makes the model behave as a standard spotter, while variable instructions filter which instances to output, and in VQA the question is encoded by the same text encoder.","core_discovery":"InstructOCR is an end-to-end encoder-decoder scene text spotter that takes both an image and a text instruction, and emits a token sequence of center points and transcriptions for the text instances that match the instruction. The paper's central discovery is that adding this linguistic conditioning—a BERT text encoder whose features are cross-attended with ResNet-50 image features—improves reading accuracy even when the instruction at test time is simply \"recognize all text.\" The paper reports consistent gains over the SPTS line of single-point spotters, with the largest lexicon-free improvements on Total-Text and ICDAR2015, and shows that instruction-based pretraining on 0.2M scene text images improves downstream scene-text VQA by 2.1 points on ST-VQA and 2.6 points on TextVQA with a 78M-parameter model. The authors present this as evidence that aligning human language instructions with visual text helps OCR-related tasks, and that the gains extend beyond spotting into question answering.","pith_inferences":["Our inference: the cleanest way to separate the instruction effect from the data effect is to retrain the SPTS baselines on the same combined pretraining set; the paper does not do this, so the dagger-row gains should be read as an upper bound on the instruction benefit.","Our inference: instruction diversity can be viewed as a freeform data augmentation over the output space; the same template idea could be applied to document parsing or key-information extraction, where attributes like font, color, or position could be encoded as instructions.","Our inference: since the VQA gain appears after instruction pretraining, instruction-following during pretraining may act as a general text-understanding regularizer, which could transfer to other OCR-heavy multimodal tasks beyond VQA."],"forward_implications":["The same architecture can serve as a controllable spotter: one model answers \"recognize all text,\" \"recognize text of five characters,\" or \"recognize text starting with A\" without retraining.","Instruction pretraining on text-spotting data carries over to VQA, improving TextVQA by 2.6 and ST-VQA by 2.1 over identical OCR-only pretraining.","Because instructions are derived from existing annotations, the method adds no data-collection cost and can be applied to any word-level scene text dataset.","The gains are achieved by a 78M-parameter model with a ResNet-50 image encoder and BERT text encoder, in contrast to the billion-parameter vision-language models used on the same VQA benchmarks.","Adding larger spotting corpora (TextOCR, HierText) to the pretraining mixture further lifts results, suggesting the mechanism scales with data."],"supporting_citations":[{"why":"Supplies the SPTS sequence-generation spotter that InstructOCR builds on and the point-based evaluation metric.","marker":"Peng, Wang et al. 2022"},{"why":"SPTS-v2, the primary point-based baseline whose lexicon-free numbers the discussion compares against.","marker":"Liu, Zhang et al. 2023"},{"why":"Bidirectional transformer whose pretrained weights initialize the instruction text encoder.","marker":"Devlin, Chang et al. 2018"},{"why":"TextOCR, one of the two extra datasets added in the InstructOCR-dagger pretraining mixture.","marker":"Singh, Pang et al. 2021"},{"why":"HierText, the other extra dataset used in the dagger pretraining mixture.","marker":"Long, Qin et al. 2022"},{"why":"UNITS, the prior method that also uses TextOCR and HierText, motivating the paper's fair-comparison retraining setup.","marker":"Kil, Kim et al. 2023"},{"why":"CLIP, whose frozen text encoder is the prior art that InstructOCR contrasts with a learnable BERT text encoder.","marker":"Radford et al. 2021"}],"fun_headline_variants":["InstructOCR: Language instructions improve scene text spotting","Instruction-based OCR model boosts scene text VQA","Human language instructions lift OCR and VQA accuracy","InstructOCR: Adding text instructions boosts text reading and VQA","Instruction pretraining lifts TextVQA by 2.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results assume that comparing InstructOCR trained with extra TextOCR and HierText data against baselines trained on their original data isolates the benefit of instructions, even though the paper does not retrain those baselines with the extra data.","fun_headline_variants_meta":{"raw":{"variants":["InstructOCR: Language instructions improve scene text spotting","Instruction-based OCR model boosts scene text VQA","Human language instructions lift OCR and VQA accuracy","InstructOCR: Adding text instructions boosts text reading and VQA","Instruction pretraining lifts TextVQA by 2.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000974,"raw_usage":{"total_tokens":4131,"prompt_tokens":928,"completion_tokens":3203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3123}},"tokens_in":544,"tokens_out":3203,"duration_ms":19841,"temperature":1.0,"reasoning_tokens":3123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:20:43.058161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the SPTS-V2 baseline with the exact pretraining recipe used for InstructOCR-dagger—including TextOCR, HierText, and the same instruction-epoch schedule—but without any instructions. If that baseline's lexicon-free Hmean on Total-Text and ICDAR2015 matches or exceeds InstructOCR-dagger, then the reported instruction gains are explained by data and training budget rather than by the instruction mechanism. Alternatively, run an ablation of InstructOCR with the text encoder removed and the same data; if it matches, the effect is not linguistic.","supporting_citations":[],"review_version":1}