REVIEW 3 major objections 5 minor 42 references
DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that a Siamese tracker can follow an object through occlusion, motion blur, and look-alike backgrounds by keeping separate external memories of the target and of the background, with an attention LSTM controlling what is…
desk verdict A sensible memory-augmented tracker with a real architectural twist, undermined by an unspecified occlusion restart rule that may break the VOT protocol and by missing code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a pair of external memory banks, one for the target's evolving appearance and one for the background, each holding $m \times m \times c$ feature maps. Writing is a weighted update $M_t = (1 - w_t) * M_{t-1} + w_t * F_{\mathrm{fore}}$ and reading is a cosine-similarity retrieval; both weights come from an LSTM controller, so the memory decides what to keep without backpropagation. The distinct piece is Eq. (7), a background-subtracted foreground readout $M = M - M_{\mathrm{back}}$, which suppresses occluders and similar-looking distractors before the heatmap is computed by convolution with the ROI feature. The second distinct piece is the memory-augmented attention: ROI features and the previous frame's foreground memory are convolved to produce attention scores, which sharpens focus and avoids the drift into background that the authors attribute to MemTrack.
What would settle it
Run DAWN as described on a video with a fully occluded interval and record the predicted bounding box every frame; if the box follows the occluder during the hidden interval, the freeze-and-resume behavior claimed in Section 3.4 is absent, and the occlusion results would need another explanation.
Extended reading notes
Core claim
The paper's central claim is that remembering the background is as important as remembering the target: the readout from foreground memory is corrected as $M = M - M_{\mathrm{back}}$ before it is correlated with the ROI feature to produce the heatmap, and this subtraction suppresses whatever background object or occluder has been memorized. The attention LSTM is also changed so that attention is generated by convolving ROI features with the foreground memory read in the previous frame, not from an LSTM hidden state. With both changes, the tracker needs no online weight updates: memory read and write is a lightweight alternative, running at 15 fps in the authors' environment. The paper's own experiments on VOT2016 and VOT2017 report this design ranking third among fast trackers and reducing failures, and DAWN-RPN shows the modules transfer to SiamRPN.
Load-bearing premise
The results for occlusion-heavy videos assume the tracker can actually tell when the target is fully hidden and when it has come back, since the method freezes the box during total occlusion; the paper never specifies how that detection works.
Editorial extensions
If this is right
- Unsupervised tracking can adapt to appearance change without per-frame backpropagation, so memory update becomes a cheap alternative to re-training weights during tracking.
- Storing background features gives a principled way to suppress occluders and look-alike objects, which is consistent with the lower failure counts the paper reports on the VOT challenges.
- The DAWN-RPN results show the modules are portable: adding dual memory and attention to SiamRPN improves it on the same VOT challenges.
- The method keeps a fixed aspect ratio and still produces competitive results, suggesting that memory and attention can substitute for elaborate scale search in some tracking pipelines.
Reading between the lines
- Editorial: because the occlusion control is specified only as freezing when total occlusion is detected, a concrete test is to log whether the predicted box ever moves during a known hidden interval; the paper leaves that detection mechanism unspecified.
- Editorial: the background-subtraction operation can be read as a general distractor-suppression layer, so it could be transplanted into other template-based trackers and tested for gains beyond Siamese backbones.
- Editorial: storing only one evolving readout per stream means long-term re-identification is not addressed; reorganizing the memory into multiple historical appearances is a natural extension the paper does not test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DAWN, an unsupervised video object tracker with a foreground and a background external memory block, together with an attention LSTM as a memory controller. The tracker is built on a SiamFC-like Siamese backbone, with an optional variant DAWN-RPN that augments SiamRPN. The authors report third place among fast trackers (fps > 10) in VOT2016 and VOT2017, and present ablations in Table 5 showing that adding background memory and the new attention scheme improves EAO over SiamFC and MemTrack. The central technical contribution is the dual-memory read/write mechanism (Eq. 7) and the memory-augmented attention (Fig. 3b).
Significance. If the architecture performs as claimed, DAWN is a lightweight, plug-in memory module that could improve Siamese trackers under occlusion, motion blur, and background distractors. The paper has genuine strengths: the architecture is clearly specified with equations, the evaluation is on external VOT benchmarks, and the ablation in Table 5 isolates the contributions of background memory and attention. The claim that DAWN-RPN directly benefits SiamRPN is potentially impactful. However, the headline occlusion-handling behavior depends on an underspecified 'detect total occlusion / freeze' rule, and the quantitative support for several claims consists of small EAO differences without statistical significance or error bars. The central contribution is defensible, but the load-bearing experimental and procedural details need to be clarified before the claims can be fully credited.
major comments (3)
- [Section 3.4, Section 1] The total-occlusion handling rule is not specified. Section 3.4 states 'When detecting total occlusion, DAWN will not update bounding box until the target re-appears in a subsequent ROI,' and Section 1 says 'we do not have sophisticated engineering other than restarting DAWN after total occlusion.' The paper never defines how total occlusion is detected, how re-appearance is recognized, what state is reset or restarted, or whether any external signal is used. This rule is load-bearing for the qualitative results on 'girl' and 'frisbee' in Table 3 and Figure 11, and for the abstract's claim of handling total occlusion. Without a concrete detector and re-detection procedure, the behavior is not reproducible, and if 'restart' implies re-initialization, it may violate the VOT unsupervised protocol where no reset is allowed. Please specify the detector, the freeze/release criterion, and the state that is reset, or remove the occlusion-handling claims.
- [Tables 1, 2, 5; Section 4.3] The reported EAO differences are small and no uncertainty or significance information is given. In Table 1, DAWN's EAO of 0.28 is below STAPLEp (0.29) and Staple (0.30); in Table 2, DAWN ties ECOhc at 0.24 and is below SiamDCF at 0.25; in Table 5, DAWN and DAWN- both have EAO 0.24. The claim of 'immediate performance gain' for DAWN-RPN in Section 4.3 is supported only by qualitative plots (Figure 8) without numerical EAO/accuracy values. Please report per-sequence scores, standard deviations, or significance tests, and give the actual EAO values for DAWN-RPN and the SiamRPN baseline so the gain can be quantified.
- [Section 4.2, Figures 6-7] The ranking claim 'ranked third in VOT2016 and VOT2017 among fast trackers running at fps > 10' depends on the unsupervised evaluation protocol. The paper does not clarify whether the freeze/restart rule for total occlusion is compatible with the VOT unsupervised setting, which forbids re-initialization, and whether any ground-truth or external detection signal is used at any point. If the rule is only a freeze, the trigger must be algorithmic and specified; if it is a re-initialization, the comparison is not protocol-equivalent. Please clarify this point, as the causal attribution of the VOT ranking to the proposed memory/attention modules is not established otherwise.
minor comments (5)
- [Section 4.3, Table 3] Table 3 uses a hand-picked set of sequences with binary success/failure marks and does not report quantitative overlap or accuracy per sequence. A full table with actual overlap scores, or a systematic selection criterion, would make the 'excellent success rate' claim in the abstract easier to verify.
- [Section 3.5 and Figure 5] The subsection is titled 'DAWN-PRN' and Figure 5 also says 'DAWN-PRN', but the method is referred to as DAWN-RPN everywhere else. Please fix the typo for consistency.
- [Equation (7)] Equation (7) writes 'M = M - Mback', but M is used earlier for the foreground memory read and also for the subtracted result. Use a different symbol for the subtracted feature map to avoid confusion.
- [Section 4.4, Figures 9-10] The attention comparison with MemTrack is presented only qualitatively. A quantitative measure, such as center location error or attention heatmap overlap, would strengthen the claim that the proposed attention produces more accurate focus.
- [Section 4.1] The hyperparameters (number of memory slots, cosine window exponent, scale smoothing factor, learning rate) are listed, but the paper does not report sensitivity to them. A brief sensitivity analysis or a statement that they were not tuned would be useful.
Circularity Check
No circularity: DAWN's empirical claims rest on external VOT benchmarks and ablations, not on definitions, fitted predictions, or a load-bearing self-citation chain.
full rationale
The paper's central claims are empirical and are validated by external VOT2016/VOT2017 benchmark comparisons and by ablations (Table 5), not by a formula that reproduces its own inputs. The memory update (Eq. 6), background-memory subtraction (Eq. 7), and heat-map prediction (Eq. 8) are constructive mechanisms whose outputs are scored against ground truth; they are not fitted quantities renamed as predictions. MAVOT [23] is a self-citation in the sense that two DAWN authors are also co-authors of MAVOT, and the background-memory idea is credited to it, but DAWN's reported gains are not derived from MAVOT and are independently tested. The underspecified 'restart after total occlusion' rule (Sec. 3.4: 'When detecting total occlusion, DAWN will not update bounding box until the target re-appears in a subsequent ROI') is a genuine reproducibility and protocol-compliance concern, and the paper itself admits 'we do not have sophisticated engineering other than restarting DAWN after total occlusion' (Sec. 1); however, this is a missing-implementation issue, not a circular derivation, because the ablation in Table 5 holds the occlusion rule fixed while varying memory and attention. No equation or claimed prediction is shown to reduce by construction to a fitted parameter, a self-citation, or a definition, so no circular step is exhibited.
Assumptions & free parameters
free parameters (5)
- number of memory slots =
8
- cosine window suppression exponent =
0.27
- target image size to bounding box scale =
1.32
- scale update smoothing factor =
0.6
- initial learning rate =
1e-4
assumptions (4)
- domain assumption VOT2016 and VOT2017 EAO and failure counts computed by the authors with the VOT toolkit faithfully represent the same protocol used for official challenge ranking.
- ad hoc to paper When total occlusion is detected, DAWN freezes the bounding box until the target re-enters the ROI, and this control rule is sufficient for the reported occlusion results.
- domain assumption Subtracting background memory from foreground memory, M = M - Mback in Eq. (7), removes distractors while retaining the target signal.
- domain assumption Pretraining on ILSVRC15 with the MemTrack protocol transfers to VOT2016 and VOT2017 with the same hyperparameters.
Cite this review
Pith. "Pith review of DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking." pith.science (2026). https://pith.science/paper/XADYSCBH
@misc{pith2026190800777,
author = {Pith},
title = {Pith review of: DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/XADYSCBH}},
note = {Machine review of arXiv:1908.00777}
}
read the original abstract
Psychological studies have found that human visual tracking system involves learning, memory, and planning. Despite recent successes, not many works have focused on memory and planning in deep learning based tracking. We are thus interested in memory augmented network, where an external memory remembers the evolving appearance of the target (foreground) object without backpropagation for updating weights. Our Dual Augmented Memory Network (DAWN) is unique in remembering both target and background, and using an improved attention LSTM memory to guide the focus on memorized features. DAWN is effective in unsupervised tracking in handling total occlusion, severe motion blur, abrupt changes in target appearance, multiple object instances, and similar foreground and background features. We present extensive quantitative and qualitative experimental comparison with state-of-the-art methods including top contenders in recent VOT challenges. Notably, despite the straightforward implementation, DAWN is ranked third in both VOT2016 and VOT2017 challenges with excellent success rate among all VOT fast trackers running at fps > 10 in unsupervised tracking in both challenges. We propose DAWN-RPN, where we simply augment our memory and attention LSTM modules to the state-of-the-art SiamRPN, and report immediate performance gain, thus demonstrating DAWN can work well with and directly benefit other models to handle difficult cases as well.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016
arXiv 2016
-
[2]
L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional siamese networks for object tracking. In ECCV, 2016
work page 2016
-
[3]
Q. Cai, Y . Pan, T. Yao, C. Yan, and T. Mei. Memory match- ing networks for one-shot image recognition. In CVPR, 2018
work page 2018
-
[4]
J. Choi, J. Kwon, and K. M. Lee. Real-time visual tracking by deep reinforced decision making. CVIU, 2018
work page 2018
-
[5]
Q. Chu, W. Ouyang, H. Li, X. Wang, B. Liu, and N. Yu. Online multi-object tracking using cnn-based single object tracker with spatial-temporal attention mechanism. In ICCV, 2017
work page 2017
-
[6]
X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang. Multi-context attention for human pose estima- tion. In CVPR, 2017
work page 2017
-
[7]
M. Danelljan, G. Bhat, F. S. Khan, M. Felsberg, et al. Eco: Efficient convolution operators for tracking. In CVPR, 2017
work page 2017
-
[8]
M. Danelljan, G. H ¨ager, F. Khan, and M. Felsberg. Accurate scale estimation for robust visual tracking. In BMVC, 2014
work page 2014
Show all 42 references
-
[9]
Danelljan, A
M. Danelljan, A. Robinson, F. S. Khan, and M. Felsberg. Beyond correlation filters: Learning continuous convolution operators for visual tracking. In ECCV, 2016
2016
-
[10]
Graves, G
A. Graves, G. Wayne, and I. Danihelka. Neural turing ma- chines. arXiv preprint arXiv:1410.5401, 2014
2014 arXiv
-
[11]
Q. Guo, W. Feng, C. Zhou, R. Huang, L. Wan, and S. Wang. Learning dynamic siamese network for visual object track- ing. In ICCV, 2017
2017
-
[12]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016
2016
-
[13]
D. Held, S. Thrun, and S. Savarese. Learning to track at 100 fps with deep regression networks. In ECCV, 2016
2016
-
[14]
J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High- speed tracking with kernelized correlation filters. TPAMI, 2015
2015
-
[15]
Hochreiter and J
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997
1997
-
[16]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015
2015
-
[17]
Kristan, A
M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, L. ˇCehovin, T. V oj´ır, G. H¨ager, A. Luke ˇziˇc, G. Fern´andez, et al. The visual object tracking vot2016 chal- lenge results. In ECCV, 2016
2016
-
[18]
Kristan, A
M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, L. C. Zajc, T. V oj ´ır, G. H¨ager, A. Luke ˇzic, A. Eldesokey, et al. The visual object tracking vot2017 chal- lenge results. In ICCVW, 2017
2017
-
[19]
Kristan, A
M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pfugfelder, L. C. Zajc, T. V ojir, G. Bhat, A. Lukezic, A. Eldesokey, G. Fernandez, and et al. The sixth visual ob- ject tracking vot2018 challenge results, 2018
2018
-
[20]
Krizhevsky, I
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012
2012
-
[21]
B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu. High perfor- mance visual tracking with siamese region proposal network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8971–8980, 2018
2018
-
[22]
H. Li, Y . Li, and F. Porikli. Deeptrack: Learning discrimina- tive feature representations online for robust visual tracking. TIP, 2016
2016
-
[23]
B. Liu, Y . Wang, Y .-W. Tai, and C.-K. Tang. Mavot: Memory-augmented video object tracking. arXiv preprint arXiv:1711.09414, 2017
2017 arXiv
-
[24]
V . Mnih, N. Heess, A. Graves, et al. Recurrent models of visual attention. In NIPS, 2014
2014
-
[25]
Nam and B
H. Nam and B. Han. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, 2016
2016
-
[26]
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015
2015
-
[27]
Russakovsky, J
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recog- nition challeng. IJCV, 2015
2015
-
[28]
Santoro, S
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065, 2016
2016 arXiv
-
[29]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015
2015
-
[30]
A. W. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah. Visual tracking: An experimental survey. TPAMI, 2014
2014
-
[31]
Srivastava, G
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014
2014
-
[32]
Sukhbaatar, J
S. Sukhbaatar, J. Weston, R. Fergus, et al. End-to-end mem- ory networks. In NIPS, 2015
2015
-
[33]
F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang. Residual attention network for image classification. CVPR, 2017
2017
-
[34]
Q. Wang, Z. Teng, J. Xing, J. Gao, W. Hu, and S. Maybank. Learning attentions: residual attentional siamese network for high performance online visual tracking. In CVPR, 2018
2018
-
[35]
Y . Wu, J. Lim, and M.-H. Yang. Online object tracking: A benchmark. In CVPR, 2013
2013
-
[36]
Y . Wu, J. Lim, and M.-H. Yang. Object tracking benchmark. TPAMI, 2015
2015
-
[37]
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhut- dinov, R. S. Zemel, and Y . Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015
2015
-
[38]
H. Yang, L. Shao, F. Zheng, L. Wang, and Z. Song. Recent advances and trends in visual tracking: A review.Neurocom- puting, 74(18):3823–3831, 2011
2011
-
[39]
Yang and A
T. Yang and A. B. Chan. Learning dynamic memory net- works for object tracking. In ECCV, 2018
2018
-
[40]
S. Yun, J. Choi, Y . Yoo, K. Yun, and J. Young Choi. Action- decision networks for visual tracking with deep reinforce- ment learning. In CVPR, 2017
2017
-
[41]
Zhang, H
D. Zhang, H. Maei, X. Wang, and Y .-F. Wang. Deep rein- forcement learning for visual object tracking in videos.arXiv preprint arXiv:1701.08936, 2017
2017 arXiv
-
[42]
Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu. Distractor-aware siamese networks for visual object track- ing. In ECCV, 2018. 9
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.