Pith. sign in

REVIEW 3 major objections 4 minor 31 references

An End-to-end Video Text Detector with Online Tracking

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that video text detection and online tracking can be trained as one end-to-end network, with ConvLSTM carrying motion memory and GRU-based descriptors associating text across frames.

desk verdict A competent incremental video-text detection paper whose claimed SOTA on YVT rests on an under-specified online tracking step; the rest of the ablations are solid but the thresholds need disclosure and sensitivity analysis. read the letter →

arxiv 1908.07135 v1 pith:VPXBA4FS submitted 2019-08-20 cs.CV

Hongyuan Yu , Chengquan Zhang , Xuan Li , Junyu Han , Errui Ding , Liang Wang This is my paper · ORCID
classification cs.CV
keywords videotextdetectiononlinetrackingConvLSTMsceneinstanceassociationappearance-geometrydescriptormulti-objectend-to-endlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that video text detection and online tracking belong in a single trainable network instead of separate stages. The detection branch uses a ConvLSTM block on shared backbone features to carry motion memory forward, while the tracking branch represents each detected text box with an appearance-geometry descriptor and uses a GRU to predict the descriptor in the next frame before matching instances into trajectories. Jointly training the two branches, the authors report F-scores about two points above the best prior numbers on ICDAR 2013 Video, Minetto, and YVT, at a real-time 24.36 frames per second on a TITAN Xp. A reader should care because it suggests temporal context and trajectory supervision help video text detection without paying for a separate tracking network.

What carries the argument

The load-bearing object is the appearance-geometry descriptor (AGD) together with its GRU-estimated next-frame version (EAGD). Appearance comes from an ROI Transform on the shared feature map followed by convolution and global pooling, giving 128 numbers. Geometry comes from embedding the eight normalized quadrangle coordinates through two fully connected layers, giving 8 numbers, and concatenation yields a 136-dimensional descriptor. A GRU consumes previous-frame descriptors, with a mask that resets the hidden state when an instance is new, and outputs predicted current-frame descriptors. These are matched to current descriptors by a similarity matrix and Kuhn-Munkres assignment with thresholds, and a contrastive loss trains positive same-instance pairs to be close and negative pairs to be far. In parallel, a ConvLSTM block on the shared features gives the detection branch long-term motion memory. The claim is that this descriptor-plus-GRU mechanism is what converts tracking from a separate post-processing stage into a nearly cost-free, jointly trainable part of detection.

What would settle it

Vary the matching threshold $\theta_m$ and the high-confidence threshold $\theta_h$ over a grid on the ICDAR 2013 Video test videos while keeping the trained network fixed. If the F-score moves by more than the claimed two-point margin across reasonable settings, the benchmark gain would undercut the claim that end-to-end training is the source of the improvement.

Watch

Extended reading notes

Core claim

The central claim is that an end-to-end video text detector with online tracking outperforms two-stage detect-then-track pipelines. In the detection branch, ConvLSTM propagates frame-level information through the shared ResNet50/U-Net features, and detection is done in EAST-like anchor-free per-pixel quadrangle regression. In the tracking branch, every text proposal is summarized by a 136-dimensional appearance-geometry descriptor, where 128 appearance dimensions come from an ROI-transformed shared feature map and 8 geometry dimensions come from embedded quadrangle coordinates. The descriptors from the previous frame pass through a GRU to produce estimated descriptors for the current frame, and a Kuhn-Munkres match between estimated and current descriptors builds trajectories. The two branches are trained together with a detection loss plus a contrastive tracking loss. With this joint model, the paper reports F-scores of 66.92 on ICDAR 2013 Video, 90.32 on Minetto, and 79.05 on YVT, exceeding their compared two-stage versions and prior methods on each benchmark.

Load-bearing premise

The reported gains assume the trajectory-generation thresholds and tracklet-length reward were fixed without per-dataset tuning; the paper does not disclose their values or a sensitivity analysis, so threshold tuning remains a possible alternative explanation for the improvement.

Editorial extensions

If this is right

  • If the central claim is correct, two-stage video text systems can be replaced by one network whose tracking branch reuses detection features, making tracking nearly free and enabling real-time operation.
  • The ConvLSTM ablation suggests temporal context is the main source of detection gain over single-frame detectors, so streams with persistent text should benefit most.
  • Joint training with the tracking loss improves detection itself, meaning trajectory supervision can serve as extra training signal for text detection.
  • Because trajectory generation is online, the method applies to streaming video without waiting for future frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The largest reported jump is in YVT precision, from 78.51 to 89.12 in end-to-end mode, while recall is nearly flat; a testable extension would be measuring whether trajectory rewards mainly suppress false positives such as background patches that look like text.
  • Since the paper notes trajectory generation is not part of training, adding a differentiable trajectory-level loss could widen the gap between the two-stage and end-to-end variants further.
  • The descriptor design only needs a region feature and a geometry vector, so the same framework should transfer to arbitrary-shape text by swapping quadrangle regression for a segmentation-based detector.
  • If the matching thresholds are disclosed and swept, a reader could separate the contribution of joint training from the contribution of inference-time trajectory reward.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an end-to-end trainable framework for video text detection with an online tracking branch. The detection branch augments a ResNet50+U-Net backbone with a ConvLSTM block to propagate temporal context, and the tracking branch builds appearance-geometry descriptors (AGD) for detected text proposals, estimates their next-frame descriptors (EAGD) with a GRU, and associates instances across frames by metric learning. At inference, an online trajectory generation algorithm (Algorithm 1) refines detection confidence scores by rewarding long matched tracklets. The method is evaluated on ICDAR 2013 Video, Minetto, and YVT, reporting state-of-the-art F-measure gains of about 2 points on each dataset and a runtime of 24.36 fps on a TITAN Xp. The paper also includes ablations of the ConvLSTM versus single-frame and Conv3D features, and of descriptor variants in a tracking metric evaluation on Minetto.

Significance. If the reported results hold, the paper would demonstrate a useful integration of video text detection and online tracking into one trainable model, with shared features and a real-time rate, which is a practically relevant contribution. The comparison of ConvLSTM against Conv3D and single-frame detection, and the ablation showing the benefit of matching AGD with EAGD, are informative. The use of public benchmarks with standard train/test splits is a positive feature; the central result is not circular in the sense of fitting a quantity and then calling it a prediction. However, the significance is currently limited by the lack of specification of the inference-time trajectory generation thresholds and the under-specification of the contrastive tracking loss, both of which are necessary to attribute the reported improvements to the proposed architecture rather than to a favorable post-processing configuration.

major comments (3)
  1. [III-D (Algorithm 1) and Table III] The online trajectory generation algorithm introduces at least five unreported settings: the detection threshold θ_l (step 1), the matching threshold θ_m (step 3), the high-confidence threshold θ_h and the reward scale τ (step 5), and the number of proposals K (Section III-B). On YVT, the end-to-end detector without online tracking reaches F=74.86 (Table III), which is below the prior state of the art (Wu et al., F=77.00); the reported F=79.05 is achieved only after applying Algorithm 1. The precision jump from 78.51 to 89.12 with essentially flat recall (71.54 to 71.03) is the characteristic signature of threshold-based suppression of low-confidence boxes rather than recovery of new true positives. Without reporting the parameter values and a sensitivity analysis, the central claim that the method significantly outperforms prior methods cannot be attributed specifically to the end-to-end architecture, ConvLSTM, AGD/GRU, or joint training. Please provide the exact values for these parameters and an experiment that shows the F-measure as a function of each threshold on at least one dataset.
  2. [Eq. (5)] The contrastive tracking loss is under-specified. The double sum over i and j with denominator K^2 implies that all pairs of the K proposals are used, but the construction of the label y (L^t_{i,j}) is not described: it is not stated how positive pairs are defined (e.g., by IoU with ground-truth trajectories), how negative pairs are sampled, whether self-pairs (i=j) are excluded, or how the number of valid proposals per frame is handled when it is less than K. Since the loss is central to the joint training claim, please specify the pair sampling procedure, the label assignment protocol, and any handling of class imbalance between positive and negative pairs. In particular, clarify whether the descriptor distance d in Eq. (5) is computed between AGD_t and EAGD_{t-1} or between raw AGDs, because Eq. (4) suggests that matching is done with the estimated descriptors, but the loss formula does not make this explicit.
  3. [Eq. (4) and Section III-C] The GRU-based descriptor estimation is not sufficiently specified to be reproducible. The notation (EAGD_t, h_t) = GRU(AGD_t, mask_t * h_{t-1}) uses a single hidden state h_{t-1}, yet in multi-target tracking there is one hidden state per text trajectory; it is unclear how hidden states are indexed, initialized, or updated for newly appearing and disappearing text instances. The mask_t is described as a scalar ('set to zero when the instance does not exist'), but the mask must be instance-specific and applied per trajectory. Without a precise description of the hidden-state bookkeeping, the trajectory generation in Algorithm 1 cannot be reimplemented. Please provide the exact state-transition equations and the procedure for creating and discarding hidden states.
minor comments (4)
  1. [Abstract and Introduction] There are typographical errors, e.g., 'palys' in the first paragraph of the Introduction, and 'rame size' in the ICDAR 2013 dataset description of Section IV-A.
  2. [Table III caption and References] The caption of Table III says 'SEVERAL PUBIC DATASETS'; 'Moslen' in the table appears to be a misspelling of Mosleh (reference [9]). Please correct these.
  3. [Section IV-C, Table I] The row 'Ours with optical flow' shows a large drop in precision (79.97 to 68.49) but an increase in recall (55.21 to 55.69) compared with 'Ours with ConvLSTM'. The paper does not explain why optical flow hurts precision so markedly; a sentence explaining this would aid the reader.
  4. [Section IV-B] The implementation details state 'Each frame contains 10 detection boxes, including positive samples and negative samples' but it is not clear how these 10 boxes are chosen during training, e.g., whether they are ground-truth boxes, predicted proposals, or a mixture. This affects the interpretation of the tracking loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SOTA claim rests on benchmark evaluations, not on a derivation that reuses its own fitted quantities.

full rationale

The paper's central claim is an empirical performance comparison on public benchmarks (ICDAR2013 Video, Minetto, YVT) using standard training/test splits. No quantity is fitted and then renamed as a prediction: the end-to-end detector and tracking branch are trained with the stated multi-task loss, and the reported F-scores are measured outputs on test data. The online trajectory generation algorithm (Algorithm 1) contains unreported thresholds (theta_l, theta_m, theta_h, tau, K), but this is a reproducibility or parameter-sensitivity concern, not circularity: the thresholds are hyperparameters that affect inference, they are not derived from the test labels or from the claimed conclusion. The only self-citation in the method is reference [15] (TextNet) for the ROI Transform layer used in the descriptor generation module; this is an architectural component and is not load-bearing for the paper's central claim of end-to-end video text detection with tracking. The matching and trajectory logic is described explicitly, and the improvements are presented as experimental results rather than as consequences of a definition. Accordingly, no step in the paper's derivation chain reduces to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard deep learning machinery and several domain assumptions about the tractability of text instance representation. The key unstated load-bearing choices are the trajectory generation thresholds and the pair-sampling procedure for the tracking loss.

free parameters (5)
  • α (detection loss weight) = 5
    Hyperparameter in Eq. 2, chosen by hand to balance classification and regression losses.
  • β (tracking loss weight) = 0.1
    Hyperparameter in Eq. 6; the paper states different values have little effect, but no ablation is shown.
  • contrastive margin m = 1.0
    Margin in Eq. 5, set by hand.
  • trajectory thresholds θm, θh, τ = unspecified
    Algorithm 1 uses these thresholds and reward; values are not reported, so the final detection F-scores depend on unstated choices.
  • proposals per frame K = 10
    Each frame contains 10 detection boxes during training; this affects the tracking loss normalization.
assumptions (5)
  • standard math Backpropagation and stochastic gradient descent function as expected for deep networks.
    Relied on for training the combined detection and tracking losses.
  • standard math Hungarian (Kuhn-Munkres) algorithm provides optimal bipartite matching for descriptor associations.
    Used in Algorithm 1 to match descriptors between frames.
  • domain assumption Text instances in video can be represented by appearance-geometry descriptors that are temporally coherent enough for GRU prediction.
    The whole tracking branch assumes such descriptors are stable across frames.
  • domain assumption The datasets (ICDAR2013 Video, Minetto, YVT) provide ground-truth tracks that can supervise the contrastive loss.
    The tracking loss requires pairs labeled as same/different instance; this label source is assumed.
  • domain assumption ConvLSTM hidden state can be propagated across frames in a video without reset issues.
    Eq. 1 assumes temporal memory is beneficial and stable; no analysis of state reset at video boundaries is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An End-to-end Video Text Detector with Online Tracking." pith.science (2026). https://pith.science/paper/VPXBA4FS

@misc{pith2026190807135,
  author       = {Pith},
  title        = {Pith review of: An End-to-end Video Text Detector with Online Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPXBA4FS}},
  note         = {Machine review of arXiv:1908.07135}
}
read the original abstract

Video text detection is considered as one of the most difficult tasks in document analysis due to the following two challenges: 1) the difficulties caused by video scenes, i.e., motion blur, illumination changes, and occlusion; 2) the properties of text including variants of fonts, languages, orientations, and shapes. Most existing methods attempt to enhance the performance of video text detection by cooperating with video text tracking, but treat these two tasks separately. In this work, we propose an end-to-end video text detection model with online tracking to address these two challenges. Specifically, in the detection branch, we adopt ConvLSTM to capture spatial structure information and motion memory. In the tracking branch, we convert the tracking problem to text instance association, and an appearance-geometry descriptor with memory mechanism is proposed to generate robust representation of text instances. By integrating these two branches into one trainable framework, they can promote each other and the computational cost is significantly reduced. Experiments on existing video text benchmarks including ICDAR2013 Video, Minetto and YVT demonstrate that the proposed method significantly outperforms state-of-the-art methods. Our method improves F-score by about 2 on all datasets and it can run realtime with 24.36 fps on TITAN Xp.

Figures

Figures reproduced from arXiv: 1908.07135 by the authors.

Figure 1
Figure 1. Video text detection and tracking. The first row is the detection [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed architecture of video text detection with online tracking. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Descriptor Generation. quality of regression offset. α is a hyper-parameter, which is set to 5 in our experiments. In addition, we use NMS (Non￾maximum suppression) to get preliminary detection results and feed top-K proposals into the next tracking branch. C. Text Tracking Branch In order to improve the robustness of text instance repre￾sentation in some difficult circumstances such as occlusion, motion blur, and e… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages

  1. [1]

    Bernardin and R

    K. Bernardin and R. Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. Journal on Image and Video Processing , 2008:1, 2008

    work page 2008

  2. [2]

    Epshtein, E

    B. Epshtein, E. Ofek, and Y . Wexler. Detecting text in natural scenes with stroke width transform. In CVPR, pages 2963– 2970, 2010

    work page 2010

  3. [3]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016

    2016

  4. [4]

    He, X.-Y

    W. He, X.-Y . Zhang, F. Yin, and C.-L. Liu. Deep direct regression for multi-oriented scene text detection. In ICCV, pages 745–753, 2017

    work page 2017

  5. [5]

    S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks for human action recognition. IEEE TPAMI, 35(1):221–231, 2013. Method ICDAR 2013 Minetto dataset YVT P R F P R F P R F Epshtein et al. [2] 39.80 32.53 35.94 − − − 68.00 76.00 72.00 Zhao et al. [29] 47.02 46.30 46.65 − − − 34.00 41.00 37.00 Minetto et al. [8] − − − 61.00 69.00 63.00 − − − Y...

    work page 2013

  6. [6]

    Karatzas, F

    D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. G. i Big- orda, S. R. Mestre, J. Mas, D. F. Mota, J. A. Almazan, and L. P. De Las Heras. Icdar 2013 robust reading competition. In ICDAR, pages 1484–1493. IEEE, 2013

    work page 2013

  7. [7]

    Khare, P

    V . Khare, P. Shivakumara, R. Paramesran, and M. Blumen- stein. Arbitrarily-oriented multi-lingual text detection in video. Multimedia Tools and Applications , 76(15):16625– 16655, 2017

    work page 2017

  8. [8]

    Minetto, N

    R. Minetto, N. Thome, M. Cord, N. J. Leite, and J. Stolfi. Snoopertrack: Text detection and tracking for outdoor videos. In ICIP, pages 505–508. IEEE, 2011

    work page 2011

Show all 31 references
  1. [9]

    Mosleh, N

    A. Mosleh, N. Bouguila, and A. B. Hamza. Automatic inpainting scheme for video text detection and removal. IEEE TIP, 22(11):4460–4472, 2013

    2013

  2. [10]

    P. X. Nguyen, K. Wang, and S. Belongie. Video text detection and recognition: Dataset and benchmark. In WACV, pages 776–783. IEEE, 2014

    2014

  3. [11]

    W.-Y . Pei, C. Yang, L.-Y . Meng, J.-B. Hou, S. Tian, and X.-C. Yin. Scene video text tracking with graph matching. IEEE Access, 6:19419–19426, 2018

    2018

  4. [12]

    Ronneberger, P

    O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCA, pages 234–241. Springer, 2015

    2015

  5. [13]

    B. Shi, X. Bai, and S. Belongie. Detecting oriented text in natural images by linking segments. In CVPR, pages 3482–

  6. [14]

    Shivakumara, L

    P. Shivakumara, L. Wu, T. Lu, C. L. Tan, M. Blumenstein, and B. S. Anami. Fractals based multi-oriented text detection system for recognition in mobile video images. Pattern Recognition, 68:158–174, 2017

    2017

  7. [15]

    Y . Sun, C. Zhang, J. Liu, J. Han, and E. Ding. Textnet: Ir- regular text reading from images with an end-to-end trainable network. In ACCV, 2018

    2018

  8. [16]

    Tian, W.-Y

    S. Tian, W.-Y . Pei, Z.-Y . Zuo, and X.-C. Yin. Scene text detection in video by learning locally and globally. In IJCAI, 2016

    2016

  9. [17]

    L. Wang, Y . Wang, S. Shan, and F. Su. Scene text detection and tracking in video with background cues. In ICMR, 2018

    2018

  10. [18]

    L. Wang, Y . Wang, S. Shan, and F. Su. Scene text detection and tracking in video with background cues. In ICMR, pages 160–168. ACM, 2018

    2018

  11. [19]

    X. Wang, Y . Jiang, S. Yang, X. Zhu, W. Li, P. Fu, H. Wang, and Z. Luo. End-to-end scene text recognition in videos based on multi frame tracking. In ICDAR, volume 1, pages 1255–

  12. [20]

    Y . Wang, L. Wang, and F. Su. A robust approach for scene text detection and tracking in video. In PCM, pages 303–314. Springer, 2018

    2018

  13. [21]

    L. Wu, P. Shivakumara, T. Lu, and C. L. Tan. A new technique for multi-oriented scene text line detection and tracking in video. IEEE Trans. Multimedia , 17(8):1137–1152, 2015

    2015

  14. [22]

    Wu and P

    Y . Wu and P. Natarajan. Self-organized text detection with minimal post-processing via border learning. In ICCV, 2017

    2017

  15. [23]

    Xingjian, Z

    S. Xingjian, Z. Chen, H. Wang, D.-Y . Yeung, W.-K. Wong, and W.-c. Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In NeurIPS, pages 802–810, 2015

    2015

  16. [24]

    X.-H. Yang, W. He, F. Yin, and C.-L. Liu. A unified video text detection method with network flow. In ICDAR, volume 1, pages 331–336. IEEE, 2017

    2017

  17. [25]

    Ye and D

    Q. Ye and D. S. Doermann. Text detection and recognition in imagery: A survey. IEEE TPAMI, 37:1480–1500, 2015

    2015

  18. [26]

    X.-C. Yin, X. Yin, K. Huang, and H.-W. Hao. Robust text detection in natural scene images. IEEE TPAMI, (1):1, 2013

    2013

  19. [27]

    Yin, Z.-Y

    X.-C. Yin, Z.-Y . Zuo, S. Tian, and C.-L. Liu. Text detection, tracking and recognition in video: A comprehensive survey. IEEE TIP, 25:2752–2773, 2016

    2016

  20. [28]

    Zhang, C

    Z. Zhang, C. Zhang, W. Shen, C. Yao, W. Liu, and X. Bai. Multi-oriented text detection with fully convolutional net- works. In CVPR, 2016

    2016

  21. [29]

    Zhao, K.-H

    X. Zhao, K.-H. Lin, Y . Fu, Y . Hu, Y . Liu, and T. S. Huang. Text from corners: a novel approach to detect text and caption in videos. IEEE TIP, 20(3):790–799, 2011

    2011

  22. [30]

    X. Zhou, C. Yao, H. Wen, Y . Wang, S. Zhou, W. He, and J. Liang. East: an efficient and accurate scene text detector. In CVPR, pages 2642–2651, 2017

    2017

  23. [31]

    Z.-Y . Zuo, S. Tian, W.-y. Pei, and X.-C. Yin. Multi-strategy tracking based text detection in scene videos. In ICDAR, pages 66–70. IEEE, 2015

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.