{"id":"9accefad-42ec-4983-b689-e438f5a710c6","arxiv_id":"2505.15649","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Scene text detectors overfit under dataset-specific fine-tuning; the paper proposes Joint-Dataset Learning and a 13-category Long-Tailed Benchmark (LTB) with a self-supervised MAEDet baseline.","lead":"This paper argues that scene text detectors overfit to narrow benchmark datasets and introduces a new long-tailed benchmark plus a joint-dataset training protocol to better measure real-world performance. It also presents MAEDet, a self-supervised baseline, and reports detailed evaluations across nine datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LTB's construction depends on an unreported IoU threshold t, so the benchmark may measure detector-failure agreement rather than the natural long-tailed distribution; the taxonomy and causal claim inherit this bias.","rationale":"The reader's weakest assumption identified the same load-bearing concern: the benchmark's selection is based on detector failures and an unspecified IoU threshold. My analysis agrees and sharpens it: the unreported t directly controls which instances enter LTB, making the benchmark's content non-reproducible and potentially unrepresentative of the natural long-tailed distribution. This is not a minor implementation detail; it affects the validity of the 13-category taxonomy, the per-category F-measures reported in Table 4, and the causal claim that long-tailed distribution is the primary cause of real-world failures. The fine-tuning gap evidence (Table 1) and the JDL protocol proposal are supported by experiments and are valuable independent contributions, so the paper should not be rejected outright. However, the benchmark's central novelty cannot be accepted unconditionally without reporting t, validating that the category distribution is stable under reasonable threshold choices, and ideally providing an independent sample of text instances to verify that the selected categories reflect actual tail frequencies. The recommended verdict remains CONDITIONAL, matching the reader's assessment, because the concern is concrete and addressable with additional experiments and transparency.","tokens_in":19286,"tokens_out":3319,"duration_ms":32422,"concrete_test":"Reconstruct LTB selection using the released source datasets with the same four detectors but several IoU thresholds (e.g., t = 0.1, 0.3, 0.5, 0.7), and recompute the category distribution from Table 7. If the relative proportions of major categories (e.g., blurred, artistic, occluded) shift by more than 20% across thresholds, the benchmark's composition is threshold-dependent and the long-tailed characterization is an artifact of the unreported t. Additionally, the authors should report the exact t used and release the selection code so the benchmark is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LTB comprehensively evaluates long-tailed scene text detection rests on the assumption that the selected 2,770 text instances are representative of the real-world long-tailed distribution. Section 4.2 and Algorithm 1 show that, for ICDAR2015, Total-Text, and ArT, instances are kept only if their maximum IoU with predictions from four detectors is below a threshold t, but t is never reported. This makes LTB a function of the failure modes of exactly those four detectors and of an unspecified hyperparameter. If t is high, LTB includes many instances that are merely difficult for these specific models; if t is low, it includes only complete misses. Either way, the benchmark's composition cannot be reproduced by other researchers, and the claim that it represents the natural long-tail is unsupported. The 13-category taxonomy in Section 4.1 is derived from the same failure-filtered set, so the taxonomy is conditioned on this selection bias. Moreover, the paper's causal statement that 'the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution' is not directly evidenced: no frequency estimates of text categories in real-world data or in training sets are provided, so 'long-tailed' is asserted rather than demonstrated. The fine-tuning gap analysis (Table 1) is a separate and more solid contribution, but the benchmark's validity is the load-bearing pillar for the paper's main novelty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that the gap between high scene-text-detection benchmark scores and poor real-world performance is caused by two factors: a fine-tuning gap from Dataset-Specific Optimization (DSO), and a long-tailed distribution of text instances with rare, challenging categories. To address the first factor, the authors propose Joint-Dataset Learning (JDL), in which models are trained and evaluated on combined training and test sets from multiple datasets. For the second factor, they conduct an error analysis of four detectors on three public test sets and use undetected instances plus manually selected hard cases to build the Long-Tailed Benchmark (LTB), containing 924 images and 2,770 challenging text instances organized into 13 categories. They also propose MAEDet, a self-supervised ViT-based detector with a CLIP-guided reconstruction loss, and report extensive comparisons on Joint98K and LTB. The paper includes code, per-category statistics, and ablations, and it makes a strong causal claim that long-tailed distribution is the primary reason for suboptimal practical performance.","tokens_in":19581,"tokens_out":5634,"duration_ms":50299,"significance":"The paper addresses a real and timely problem: academic benchmarks for scene text detection have become saturated while deployed detectors still fail on hard instances. The JDL protocol is a sensible evaluation alternative and the fine-tuning gap is a plausible phenomenon, supported by Table 1 for two detectors. The LTB benchmark, if constructed reliably, would be a useful diagnostic resource, and the authors provide a concrete taxonomy, explicit per-category counts in Table 7, and a public code repository. The proposal of MAEDet as a self-supervised baseline is also a constructive contribution. However, the significance depends critically on whether LTB truly represents the natural long-tailed distribution rather than the failure modes of the four specific detectors used to filter it; this is exactly where the current manuscript is weakest.","major_comments":[{"comment":"The IoU threshold t in Algorithm 1 is never reported, so the composition of LTB cannot be reproduced. This is not a stylistic omission: t determines whether an instance is 'undetected by all detectors' or merely partially matched, and different choices of t will change both the instance set and the category distribution of LTB. Please report t, justify its value, and provide a sensitivity analysis showing how LTB contents and category counts change over a plausible range of t.","section":"Section 4.2 / Algorithm 1"},{"comment":"LTB instances from ICDAR2015, Total-Text, and ArT are selected as ground-truth boxes whose maximum IoU with predictions from DBNet++, ABCNet v2, DPText-DETR, and SRFormer is below t, and the 13-category taxonomy is derived from these failure-filtered cases. The benchmark therefore measures agreement among the failure modes of those four detectors, not necessarily the natural long-tailed distribution of scene text. The claim that LTB provides 'a credibility benchmark' for real-world detection requires evidence that the selected instances are representative, for example frequency estimates of the 13 categories in unfiltered real-world images, or a demonstration that category distributions are stable across different detector ensembles and thresholds.","section":"Section 4.2 / Appendix C.2"},{"comment":"Joint models are trained on Joint98K, which contains the training splits of ICDAR2015, Total-Text, and ArT, while LTB is constructed from test instances of exactly those same datasets (plus manual additions). Table 4 therefore evaluates joint models on hard instances from datasets whose training distributions were seen during training; this is an in-distribution hard-example test, not an out-of-distribution test of real-world generalization. To support the claim that JDL improves long-tailed robustness, the authors should evaluate LTB on models trained without access to the training splits of the source datasets, or explicitly state the in-distribution nature of the evaluation and its implications.","section":"Section 5.2 / Table 4 and Appendix C.1"},{"comment":"The abstract's causal statement that 'the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution' is not directly evidenced. The paper shows that four detectors miss some instances and that the filtered set has uneven category counts, but it does not measure the frequency of text categories in real-world data, does not estimate category frequencies in training sets, and does not compare the long-tail factor against other potential causes such as domain shift, resolution, or annotation noise. Please soften this to a contributing factor, or add direct evidence such as a training-set frequency analysis and a comparison of failure rates across categories.","section":"Abstract and Section 1"},{"comment":"The hyperparameters of MAEDet (input size, mask threshold T, and balance factor alpha) are selected by directly evaluating on the LTB test set. Tuning the proposed baseline on the benchmark it is meant to support invalidates the reported baseline numbers and creates an unfair baseline for future comparisons. Please use a held-out validation split of LTB for hyperparameter selection, or clearly disclose and justify the test-set tuning.","section":"Section 5.3 / Table 5"}],"minor_comments":[{"comment":"The text says '1,012 occluded text instances (Column 1) and 960 blurred text instances (Column 10)', but Table 7 shows Column 1 is blurred (962 total) and Column 10 is occluded (1,012 total); the column references are reversed.","section":"Appendix C.2 / Table 7"},{"comment":"The sentence 'MAEDet includes tree components' contains a typo; it should read 'three components'.","section":"Appendix D.2"},{"comment":"Algorithm 1 and the filtering process appear only in the appendix; the main text refers to the filtering conceptually but does not cite Algorithm 1. Moving the algorithm to the main text or adding an explicit reference would improve readability and reproducibility.","section":"Section 4.2 and Appendix C.2"},{"comment":"The subsection 'Similar Protocols in Other Deep Learning Domains' mentions 'Wang et al.' and 'He et al.' without citations; please add the corresponding references.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-construction issue is the central risk: the unreported IoU threshold and the detector-conditional selection make the claimed representativeness of LTB unverifiable. The causal claim about long-tailed distribution also needs stronger evidence or softer wording. These issues are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look if you work on scene text detection or benchmark design. The paper's real contribution is LTB: 924 images, 2,770 challenging text instances organized into 13 categories across three major families (intra-instance, inter-instance, background). That alone is useful—it's the first benchmark that puts several known long-tailed challenges (artistic, inverse, dense, occluded, low-contrast) into one evaluation platform. The quantitative failure analysis (detectors fail on 7% of instances, 65% of those are genuinely challenging) is a nice, concrete piece of evidence. The fine-tuning gap demonstration in Table 1—showing that ABCNet v2 and DPText drop by up to 8.8 F-measure when fine-tuned on one dataset and evaluated on another—is solid, even if the phenomenon is known in the domain-adaptation literature they cite.\n\nThe soft spots are real but fixable. The biggest is in Section 4.2/Algorithm 1: the IoU threshold t that decides which ground truth instances are kept in LTB is never reported. That means the construction can't be reproduced, and the claim that LTB represents the natural long-tailed distribution is not really supported—the benchmark is selected from the failure modes of four specific detectors. It probably still functions as a hard-case stress test, but the 'natural long-tail' framing needs to be toned down or defended with a sensitivity analysis over t. Relatedly, the abstract says suboptimal performance in practice is 'primarily attributed' to long-tailed distribution, but no frequency estimates of text categories in the wild or in training sets are given. That's an overreach. The JDL protocol is reasonable, but it's not deeply novel; its value is the unified evaluation, which the paper demonstrates. Finally, MAEDet's hyperparameters (T, alpha) are chosen based on LTB performance in the ablation (Table 5), so the baseline is fitted to the very benchmark it's supposed to evaluate. A held-out validation split would fix that.\n\nOverall: the benchmark and protocol are worth engaging with. I'd send it to review and ask for the threshold, a sensitivity check, a validation split for the baseline, and a more careful causal claim. The fine-tuning gap results alone justify referee time, and LTB could become a standard testbed if the release is verifiable.\n\nRecommendation: accept with revisions, conditional on those items.","headline":"A genuinely useful long-tailed scene text benchmark, but the unreported IoU threshold in its construction and the overstated causal claim need fixing before I'd trust the central narrative.","tokens_in":20113,"tokens_out":2897,"would_cite":true,"duration_ms":25715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scene-text detector scores on academic benchmarks are inflated by per-dataset fine-tuning and by a long tail of rare text types that benchmarks ignore.","keywords":["scene text detection","long-tailed distribution","benchmark construction","dataset-specific optimization","joint-dataset learning","self-supervised learning","masked autoencoder","fine-tuning gap"],"falsifier":"Collect an independent sample of rare-text images without using the four detectors for selection, score the same models on it, and compare the ranking to LTB; if the ranking flips, LTB tracks detector agreement rather than intrinsic difficulty. A cheaper check is to recompute LTB membership across a sweep of the unreported IoU threshold $t$ and see whether the 13-category distribution and model ordering remain stable.","tokens_in":19091,"feed_emoji":"🔤","tokens_out":12296,"duration_ms":88050,"temperature":0.7,"pith_summary":"The paper argues that the gap between scene-text detectors' near-saturation scores on academic benchmarks and their uneven behavior in practice has two analyzable sources. The first is the Fine-tuning Gap: under Dataset-Specific Optimization (DSO), a model is fine-tuned on one dataset's training split and tested on that same dataset's test split, so benchmark scores reward overfitting to dataset-specific biases. The second is the long-tailed distribution of real-world text, where rare but consequential categories such as artistic, inverse, overlapped, occluded, and low-contrast text are under-represented in training data and in existing benchmarks. To address both, the paper proposes Joint-Dataset Learning (JDL), training and testing on the union of many datasets, and introduces the Long-Tailed Benchmark (LTB), 924 images with 2,770 annotated challenging instances grouped into 13 subcategories under intra-instance, inter-instance, and background problem families. If the diagnosis holds, published F-measures from per-dataset fine-tuning overstate generalization, and LTB gives the field a way to measure progress on the cases that actually break detectors in deployment.","feed_headline":"Benchmark scores hide real-world failures in scene-text detectors","feed_subtitle":"A 924-image long-tail benchmark and joint-training protocol reveal why detectors stumble on rare text.","key_machinery":"The argument runs on three mechanisms. The Fine-tuning Gap is operationalized by comparing F-measure on a target dataset before and after fine-tuning, and by testing a model fine-tuned on one dataset against another dataset's test set. The Joint-Dataset Learning protocol replaces per-dataset fine-tuning with training on the union $\\bigcup_i D_i^{\\text{train}}$ and evaluating on $\\bigcup_i D_i^{\\text{test}}$, applied in the paper to Joint98K, a union of nine datasets with 98,015 training images and 21,675 test images. The Long-Tailed Benchmark is built by Algorithm 1, which keeps ground-truth instances whose maximum Intersection-over-Union with any prediction from four detectors falls below an unreported threshold $t$, plus manual selection from InverseText, Occluded-RoadText, and NightTime-ArT; these instances are organized into three groups (intra-instance, inter-instance, background) and 13 subcategories. Finally, MAEDet pairs a Masked Auto-Encoder backbone with a frozen CLIP attention map that biases reconstruction toward text pixels through a balanced reconstruction loss $\\mathcal{L}_{br} = \\sum_{i,j} \\left(\\alpha \\mathbf{1}_{M_g > T} + (1-\\alpha)\\mathbf{1}_{M_g \\le T}\\right) L_2(I_{ij}, I^{\\text{rec}}_{ij})$.","core_discovery":"The paper's central claim is that the discrepancy between near-saturated academic benchmark results and disappointing real-world behavior in scene text detection has two named causes. The first is the Fine-tuning Gap: under Dataset-Specific Optimization, a model is pretrained broadly and then fine-tuned on one dataset's training split, so its test-set score reflects overfitting to that dataset's biases; the paper reports fine-tuning gains of at least 2.0 F-measure points on the target dataset alongside drops of up to 8.8 points when the same recipe is evaluated on another dataset. The second cause is the long-tailed distribution of text itself: a failure analysis of four detectors on three datasets finds that they miss 7% of text instances, and only 65% of those misses are genuinely hard, with the remainder split between annotation errors and non-Latin scripts; rare categories such as artistic, inverse, overlapped, occluded, and low-contrast text dominate the hard failures. To correct the first cause, the paper advocates Joint-Dataset Learning (JDL), training on the union of datasets and evaluating on the union of test sets, which it shows reduces cross-dataset performance variance. To correct the second, it introduces LTB, a 924-image benchmark whose 2,770 challenging instances are labeled into 13 subcategories under intra-instance, inter-instance, and background problem families, and contributes MAEDet, a self-supervised baseline that reconstructs masked images with a CLIP-derived text-region mask before supervised detection training. If the claim is right, published DSO-era scores overstate generalization and LTB is a diagnostic instrument for the field.","pith_inferences":["Extension: because LTB selects failures by consensus of four detectors, an independent test would sample rare-text images with a wholly different detector family; if rankings invert, LTB would be measuring shared blind spots rather than intrinsic long-tail difficulty.","Extension: the three-group, 13-subcategory taxonomy is generic enough to transfer to other fine-grained detection tasks, such as traffic-sign or packaging-text detection, where per-dataset fine-tuning similarly hides rare categories; the paper does not make this claim.","Extension: publishing the IoU threshold $t$ from Algorithm 1 and a sensitivity curve would let users calibrate how hard LTB is, turning the benchmark into a tunable instrument as detectors improve.","Extension: a direct deployment check would be to run cameras or document pipelines on freshly collected scenes, compare failure rates on tail text with LTB Hard scores, and see whether LTB predicts which models fail in the field."],"forward_implications":["Published F-measures from the Dataset-Specific Optimization protocol mix real generalization with benchmark-specific overfitting, so rankings produced under DSO should be re-read as upper bounds rather than deployment predictions.","Moving to Joint-Dataset Learning gives a more transferable ranking at no algorithmic cost: the paper reports that cross-dataset variance for DPText-DETR drops from 15.1 to 6.0 F-measure points under JDL.","LTB's Hard metric, which scores only the 2,770 challenging instances, separates models on long-tail ability even when their Norm scores over the full annotations look similar.","Because the average LTB instance carries about 1.5 challenge attributes, methods tuned to a single failure mode will saturate on isolated benchmarks while still failing on compound cases.","Text-aware self-supervised pretraining of the MAEDet type raises detection on tail categories, which suggests that data-side solutions can complement architecture changes in long-tailed text detection."],"supporting_citations":[{"why":"Supplies ICDAR2015, a source of failure cases for LTB and the F-measure protocol used throughout.","marker":"[Karatzas et al., 2015]"},{"why":"Supplies Total-Text, a curved-text dataset used in failure analysis, fine-tuning-gap experiments, and LTB.","marker":"[Ch'ng and Chan, 2017]"},{"why":"Supplies ArT, the arbitrary-shaped-text dataset used in failure analysis, fine-tuning-gap experiments, and LTB.","marker":"[Chng et al., 2019]"},{"why":"Provides DPText-DETR, one of the four detectors in the failure analysis, and InverseText, one of the manual-selection sources for LTB.","marker":"[Ye et al., 2023]"},{"why":"Provides ABCNet v2, one of the four detectors used in the failure analysis and fine-tuning-gap experiments.","marker":"[Liu et al., 2021]"},{"why":"Provides SRFormer, the fourth detector used in LTB failure analysis and long-tail evaluation.","marker":"[Bu et al., 2024]"},{"why":"Supplies the Masked Auto-Encoder pretraining mechanism that MAEDet builds on.","marker":"[He et al., 2022]"},{"why":"Supplies the frozen CLIP text-region attention mask that guides MAEDet's reconstruction toward text pixels.","marker":"[Radford et al., 2021]"},{"why":"Supplies the data-perspective, self-supervised rationale and the MAERec inspiration for MAEDet.","marker":"[Jiang et al., 2023]"},{"why":"Supplies NightTime-ArT, a low-light dataset manually selected into LTB for long-tailed background challenges.","marker":"[Yu et al., 2023]"}],"fun_headline_variants":["Benchmarks lie: text detectors fail on rare, real-world text","Fine-tuning gap and long-tailed text: why detectors stumble","Joint training plus new 924-image benchmark reveal long-tail failures","Why text detectors ace benchmarks but miss artistic, overlapped text","New benchmark and self-supervised baseline target rare scene text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the text instances missed by the four chosen detectors on three source datasets, plus manually selected hard cases from three other datasets, form a representative sample of the real-world long tail; the unreported IoU threshold $t$ in the filtering algorithm is part of this premise.","fun_headline_variants_meta":{"raw":{"variants":["Benchmarks lie: text detectors fail on rare, real-world text","Fine-tuning gap and long-tailed text: why detectors stumble","Joint training plus new 924-image benchmark reveal long-tail failures","Why text detectors ace benchmarks but miss artistic, overlapped text","New benchmark and self-supervised baseline target rare scene text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000892,"raw_usage":{"total_tokens":4264,"prompt_tokens":1156,"completion_tokens":3108,"prompt_tokens_details":{"cached_tokens":1024},"prompt_cache_hit_tokens":1024,"prompt_cache_miss_tokens":132,"completion_tokens_details":{"reasoning_tokens":3023}},"tokens_in":132,"tokens_out":3108,"duration_ms":22512,"temperature":1.0,"reasoning_tokens":3023,"cache_read_input_tokens":1024,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:12:42.656840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect an independent sample of rare-text images without using the four detectors for selection, score the same models on it, and compare the ranking to LTB; if the ranking flips, LTB tracks detector agreement rather than intrinsic difficulty. A cheaper check is to recompute LTB membership across a sweep of the unreported IoU threshold $t$ and see whether the 13-category distribution and model ordering remain stable.","supporting_citations":[{"cited_title":"Ic- dar 2015 competition on robust reading","cited_arxiv_id":null,"evidence_quote":"Supplies ICDAR2015, a source of failure cases for LTB and the F-measure protocol used throughout."},{"cited_title":"Total-text: A comprehensive dataset for scene text detection and recognition","cited_arxiv_id":null,"evidence_quote":"Supplies Total-Text, a curved-text dataset used in failure analysis, fine-tuning-gap experiments, and LTB."},{"cited_title":"Ic- dar2019 robust reading challenge on arbitrary-shaped text- rrc-art","cited_arxiv_id":null,"evidence_quote":"Supplies ArT, the arbitrary-shaped-text dataset used in failure analysis, fine-tuning-gap experiments, and LTB."},{"cited_title":"Abcnet v2: Adaptive bezier-curve network for real-time end-to- end text spotting","cited_arxiv_id":null,"evidence_quote":"Provides ABCNet v2, one of the four detectors used in the failure analysis and fine-tuning-gap experiments."},{"cited_title":"Revisiting scene text recognition: A data perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the data-perspective, self-supervised rationale and the MAERec inspiration for MAEDet."}],"review_version":1}