REVIEW 2 major objections 4 minor 28 references
Saccader: Improving Accuracy of Hard Attention Models for Vision
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Hard attention can be trained at ImageNet scale when the glimpse policy is pretrained to imitate the ordering of per-location class logits, reaching 75% top-1 while viewing less than one-third of the image.
desk verdict Saccader delivers a genuine hard-attention result at ImageNet scale with a useful pretraining trick, but the '29.47% of the image' headline is glimpse union coverage, not actual pixels processed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the three-component Saccader architecture plus its two-stage training schedule. The representation network is a BagNet-77-lowD CNN whose restricted $77\times77$ receptive field yields 2048-dimensional features and 1000-dimensional logits at 361 image locations; the attention network combines these “what” features with wider-field “where” features; and the Saccader cell converts the mixed features into a spatial-softmax policy over locations while a state mask prevents revisits. The training mechanism is the pretraining objective of Eq. (5): locations sorted by descending maximum class logit are used as target glimpses for the location network, after which Eq. (6) applies REINFORCE with a baseline and two Monte Carlo trajectories to maximize the probability of a correct final prediction. The sorted-logit targets are what turn a sparse, reward-only signal into a dense supervisory signal.
What would settle it
Retrain the identical architecture with the sorted-location pretraining targets replaced by random permutations of locations, keeping the REINFORCE stage unchanged; if six-glimpse top-1 accuracy stays near 75.03%, sorted-logit pretraining is not the load-bearing ingredient.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a hard attention policy can be made trainable on a large, diverse classification benchmark by initializing it from an auxiliary task that requires no location annotations. The representation network, BagNet-77-lowD, has a $77\times77$ receptive field and computes features and class logits at 361 locations in a single pass; ordering these locations by their maximum logit gives a target glimpse sequence, and the location network is pretrained to emit that sequence before the full model is trained with REINFORCE on a reward that is 1 when the final averaged-logits prediction is correct. In the strongest configuration, a NASNet classifier fine-tuned on the selected patches reaches $75.03\pm0.08\%$ top-1 and $91.19\pm0.22\%$ top-5 with six glimpses while processing $29.47\pm0.26\%$ of the image with the NASNet. The paper also reports two boundaries: the attention network has access to the entire image, so the selection mechanism itself is not locally interpretable, and hard attention does not by itself provide adversarial robustness.
Load-bearing premise
The whole method depends on the assumption that the image regions giving the largest class scores in a network trained only on image labels are also the regions where the class evidence lives; if that link is weak, the pretraining hands the attention policy a poor starting point and policy gradient must rediscover good locations from a sparse reward.
Editorial extensions
If this is right
- Hard attention is no longer confined to simple benchmarks: the class-label-only recipe reaches 75% top-1 and 91% top-5 on ImageNet while processing under a third of the image.
- The learned policy transfers across sequence length: Saccader accuracy keeps rising when evaluated with more glimpses than the six used in training, whereas the DRAM baseline does not benefit from additional glimpses.
- The selected glimpses carry class evidence in a testable sense: occluding Saccader-selected patches causes a larger drop in a ResNet-v2-50 classifier than occluding random, edge-based, logits-ordered, or DRAM-selected patches.
- Decoupling the classifier from the localizer helps: attaching a NASNet classifier and applying it to higher-resolution $331\times331$ patches raises accuracy beyond the single-network Saccader, so the where and what components can be scaled separately.
Reading between the lines
- If the logit-ordering pretraining is the real engine, the recipe should transfer across domains: any pretrained restricted-field classifier could supply ordered location targets, so hard attention may become trainable on medical or satellite imagery with only image-level labels.
- A direct test of the logits-locate-evidence premise is to compare Saccader's glimpse sequence with human eye-tracking fixations on the same images; high agreement would suggest the policy learns a general saliency function rather than a classifier-specific one.
- Since accuracy is reported as a function of area covered, one could modify the Saccader cell to stop glimpsing once a confidence threshold is met, turning the coverage fraction into an explicit compute budget.
- The paper's negative adversarial-robustness result implies that simply restricting the classifier's input is not a defense; a natural follow-up is to feed Saccader glimpses into an adversarially trained classifier and test whether the attention policy preserves that robustness.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Saccader, a hard visual attention model for ImageNet-scale classification. The architecture combines a BagNet-77-lowD representation network with a restricted receptive field, an attention network with a wide receptive field, and a recurrent Saccader cell that maintains a visited-location state. Training proceeds in three stages: pretraining the representation network with an average-logits cross-entropy objective, self-supervised pretraining of the location-selection components to imitate the ranking of per-location logits (Eq. 5), and REINFORCE fine-tuning of the full model with a baseline and sampled trajectories. On ImageNet, the best Saccader-NASNet model reaches 75.03% top-1 and 91.19% top-5 accuracy with six glimpses whose union covers 29.47% of the image, and occlusion of the selected patches substantially impairs a ResNet classifier. The appendix additionally reports an honest negative result: hard attention does not confer adversarial robustness.
Significance. If the results hold, this is a valuable step toward training hard attention models on complex, large-scale datasets. The paper gives a concrete pretraining recipe that makes policy-gradient training tractable at ImageNet scale, supports it with ablations (no pretraining, no attention network), reports error bars over five random initializations, and compares against several attention policies including DRAM, random, edge-based, and ordered-logits baselines. The occlusion analysis connects selected glimpses to class-relevant evidence, and the adversarial-robustness appendix is a useful negative finding. The interpretability claim is appropriately partial: the classification path is restricted to the selected patches, but the attention network and the representation network observe the full image, as the conclusion acknowledges.
major comments (2)
- [Section 4.3 and abstract] The statement that Saccader-NASNet achieves 75.03% top-1 accuracy 'while processing only 29.47±0.26% of the image with the NASNet' conflates the union of the selected glimpse regions with the amount of computation actually performed. The NASNet is applied independently to each of the six 113×113 patches, so the total patch area processed is roughly 6×113²/331² ≈ 70% of the image, not 29.47%; overlapping pixels are processed multiple times. In addition, the base Saccader representation network and the attention network process the full input image (Section 3.1) even before the NASNet stage. The accuracy result itself is not in question, but the efficiency and 'attending to less than one-third of the image' framing overstates the reduction in processed pixels. The paper should state that the final classification averages logits extracted from patches whose union covers 29.47% of the image, and if computational savings are claimed, it should report the total pixel or FLOP cost of the full pipeline.
- [Section 3.2, Eq. (5)] The pretraining targets in Eq. (5) are the sorted order of per-location maximum logits of the BagNet-77-lowD representation network, which is itself trained only on class labels; the location signal is therefore self-generated rather than externally grounded. The no-pretraining ablation (Supp. Fig. 3) shows that this pretraining stage is helpful, but it does not isolate whether the logit-ranking proxy is the reason, as opposed to any structured initialization that helps REINFORCE. Since the final policy is initialized from this pretrained policy, the method's generality depends on the assumption that per-patch logits of a bag-of-local-features model localize class-discriminative evidence. The paper should explicitly state and ideally test this assumption, for example by measuring the overlap between logit-ordered locations and the final Saccader-selected locations, or by comparing against an alternative proxy-based initialization.
minor comments (4)
- [Section 3.1, Eqs. (1) and (3)] The masking term is rendered as '−105 C^{t-1}_{ij}' in the equations and should be '−10^5 C^{t-1}_{ij}' to avoid being read as minus one hundred five.
- [Section 3.2, Eq. (5)] The number of pretraining glimpses T appears in the objective before it is defined; state T = 12 in the text immediately preceding or following the equation.
- [Appendix D, Table Supp.6] The clean accuracies in the adversarial-robustness table are reported on a 3906-image subset; state this explicitly in the table caption as well as in the text.
- [Figure 3] The caption reports error bars from 5 random initializations, but it is not clear whether the black base-network markers also use error bars or are single runs; clarify in the caption.
Circularity Check
No significant circularity: the self-generated pretraining targets are an initialization heuristic, while the central accuracy claims are measured against external class labels and independent benchmarks.
full rationale
The paper's location-network pretraining objective in Eq. (5) uses target locations sorted by the maximum logits of the representation network, so the pretraining signal is generated by the model's own BagNet-77-lowD backbone. This is self-referential as an initialization scheme, but it is not a circular derivation: the pretraining is explicitly labeled as 'self-supervision' whose purpose is to alter the training trajectory, and the final policy is optimized in Eq. (6) with a REINFORCE reward based on whether the final classification is correct using only class labels. The headline accuracies are reported on the held-out ImageNet validation set after this full training procedure, so they are measured empirical results rather than quantities forced by the pretraining targets. The paper also provides independent evidence: a no-pretraining ablation (Supp. Fig. 3), comparisons against random, edge-based, ordered-logits, and DRAM policies, and an occlusion analysis using a separately pretrained ResNet-v2-50 (Fig. 5b). The concern that 'processing only 29.47% of the image with the NASNet' may overstate computational savings is a question of how computation is counted for overlapping extracted patches, not a circularity in the derivation. Self-citations in the paper, such as Kornblith et al. 2019 for transferability and Zoph et al. 2018 for NASNet, are contextual or architectural and are not load-bearing for the central attention-training claim. No step in the paper reduces a prediction to its own input by construction.
Assumptions & free parameters
free parameters (6)
- Number of glimpses T for REINFORCE training =
6
- Location pretraining sequence length =
12
- Visited-location mask constant =
-1e5
- Bottleneck dimension for what features =
512
- L2 regularization weight lambda =
8e-5
- L2 regularization weight nu =
8e-5
assumptions (4)
- domain assumption Class labels alone provide enough signal to train a useful hard attention policy.
- domain assumption Per-location maximum logits from the representation network rank locations by task relevance.
- domain assumption A 77x77 receptive field is sufficient for useful ImageNet classification.
- domain assumption Policy gradient with 2 Monte Carlo samples and a baseline has manageable variance.
invented entities (1)
-
Saccader cell with visited-location state
Cite this review
Pith. "Pith review of Saccader: Improving Accuracy of Hard Attention Models for Vision." pith.science (2026). https://pith.science/paper/PR374HDW
@misc{pith2026190807644,
author = {Pith},
title = {Pith review of: Saccader: Improving Accuracy of Hard Attention Models for Vision},
year = {2026},
howpublished = {\url{https://pith.science/paper/PR374HDW}},
note = {Machine review of arXiv:1908.07644}
}
abstract
Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret. One approach that offers some level of interpretability by design is \textit{hard attention}, which uses only relevant portions of the image. However, training hard attention models with only class label supervision is challenging, and hard attention has proved difficult to scale to complex datasets. Here, we propose a novel hard attention model, which we term Saccader. Key to Saccader is a pretraining step that requires only class labels and provides initial attention locations for policy gradient optimization. Our best models narrow the gap to common ImageNet baselines, achieving $75\%$ top-1 and $91\%$ top-5 while attending to less than one-third of the image.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[8]
URL http://arxiv.org/abs/1806.10758. J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141,
- [9]
-
[10]
M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pages 2017–2025,
work page 2017
-
[11]
P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867,
-
[12]
Y . Luo, X. Boix, G. Roig, T. Poggio, and Q. Zhao. Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv:1511.06292,
- [13]
-
[15]
E. Real, A. Aggarwal, Y . Huang, and Q. V . Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548,
-
[16]
Attention for Fine-Grained Categorization
URL http://arxiv. org/abs/1412.7054. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,
Show all 28 references
-
[19]
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y . Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057,
-
[20]
Yosinski, J
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579,
-
[22]
14 (a) (b) Figure Supp.2: Top-5 accuracy vs
13 Appendix A Supplementary Figures Wine bottle Vending machineSea slug Scuba diver Tow truck PomeranianBull mastif Partridge Windsor tie GyromitraKing snake Cougar (label: Neck brace) VasePay phone Sea urchin Packet Indian cobraSki Shoe shop (label: Garbage truck) (label: Cof...
-
[24]
** Second stage of classification weights pretraining with a narrow receptive field
Model learning rate batch size epochs λ ν DRAM (pretraining 200× 200)* 0.8 2048 120 8× 10−5 N/A DRAM (pretraining 77× 77)** 0.001 2048 120 8× 10−5 N/A DRAM*** 0.01 1024 120 8× 10−5 0 * First stage of classification weights pretraining with a wide receptive field. ** Second stage...
-
[25]
and adapted changes similar to those proposed by Sermanet et al. [2015]. In particular, the model consists of a powerful CNN (here we used ResNet-v2-50) that process a multi resolution crops concatenated along channels. The high resolution crop has the smallest receptive field,...
2015
-
[28]
This hypothesis is attractive because it provides a natural explanation for the apparent robustness of human vision to adversarial examples
previously suggested that foveation-based vision mechanisms enjoy natural robust- ness to adversarial perturbation. This hypothesis is attractive because it provides a natural explanation for the apparent robustness of human vision to adversarial examples. However, no attentio...
2018
-
[29]
It is possible that additional robustness could be achieved by using a stochastic rather than determin- istic attention mechanism, as implied by Luo et al
achieves 42.6% accuracy with a perturbation ofϵ = 16, and their adversarial training baseline achieves 39.2%. It is possible that additional robustness could be achieved by using a stochastic rather than determin- istic attention mechanism, as implied by Luo et al. [2015]. How...
2015
-
[77]
[2015]) with the change of averaging the logits to compute one cross entropy loss instead of having multiple cross entropy losses for each combination of views
Each stage combines all the different glimpses (similar to Figure 4 Sermanet et al. [2015]) with the change of averaging the logits to compute one cross entropy loss instead of having multiple cross entropy losses for each combination of views. During pretraining, we unrolled ...
2015
-
[224]
Table Supp.3: CNNs hyperparameters (ImageNet 331×
Model learning rate batch size epochs λ ResNet-v2-50 0.9 2048 120 8× 10−5 BagNet-77 0.5 2048 120 8× 10−5 BagNet-77-lowD 0.5 2048 120 8× 10−5 NASNet 1.6 4096 156 8× 10−5 NASNet-77* 0.001 1024 120 8× 10−5 * Fine tuning starting from a trained NASNet using crops of size 77× 77 id...
-
[1024]
The output of the LSTM classification layer is then fed to another LSTM layer of size 1024 for location prediction. The output of the location LSTM is passed to fully connected layer of 1024 units with ReLU activation, then passed to a 2D fully connected layer with tanh activat...
2015
-
[1988]
N. J. Butko and J. R. Movellan. I-pomdp: An infomax model of eye movement. In 2008 7th IEEE International Conference on Development and Learning, pages 139–144. IEEE,
2008
-
[1992]
J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806,
-
[1993]
Papernot, F
N. Papernot, F. Faghri, N. Carlini, I. Goodfellow, R. Feinman, A. Kurakin, C. Xie, Y . Sharma, T. Brown, A. Roy, A. Matyasko, V . Behzadan, K. Hambardzumyan, Z. Zhang, Y .-L. Juang, Z. Li, R. Sheatsley, A. Garg, J. Uesato, W. Gierke, Y . Dong, D. Berthelot, P. Hendricks, J. Ra...
-
[2000]
Szegedy, W
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199,
-
[2010]
Bahdanau, K
D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473,
-
[2014]
Gregor, I
K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623,
-
[2016]
L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595,
-
[2017]
I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572,
-
[2018]
URL http://proceedings.mlr.press/v80/ athalye18b.html
PMLR. URL http://proceedings.mlr.press/v80/ athalye18b.html. J. Ba, V . Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention.arXiv preprint arXiv:1412.7755,
-
[2019]
URL https://openreview.net/forum?id=SkfMWhAqYQ. P. J. Burt. Attention mechanisms for vision in a dynamic world. In [1988 Proceedings] 9th International Conference on Pattern Recognition, pages 977–987. IEEE,
1988
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.