REVIEW 3 major objections 5 minor 25 references
On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that Iterative Patch Selection fails to generalize below an object-to-image threshold that depends on training data size and task, and that in low-data settings a patch size smaller than the region of interest recovers…
desk verdict Useful empirical study with a real confound in the central O2I experiment; the patch-size findings and testbed are worth keeping, but the headline claim needs a cleaner experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is Iterative Patch Selection: the image is split into patches, batches are encoded and scored by a no-gradient cross-attention module, only the top M most salient patches are kept in memory after each iteration, and the survivors are aggregated by an attention-weighted mean. The testbed couples this module with a fixed-canvas megapixel MNIST variant whose object-to-image ratio changes by scaling digit size, and a Bezier-curve noise generator whose control-point counts and line thickness mimic digits. Patch size, training set size, and O2I ratio are the controlled variables that expose the generalization boundary.
What would settle it
Run the same 3000 by 3000 megapixel MNIST setup with a fixed number of noise components, say 800, across digit sizes 28, 56, 84, and 112 while keeping noise thickness fixed; if the accuracy-versus-dataset-size pattern disappears, the reported O2I threshold is a noise-density effect rather than an object-size effect.
Extended reading notes
Core claim
Iterative Patch Selection, a weakly supervised transformer-based classifier that keeps only the top M salient patches, has a generalization threshold in object-to-image ratio: below a certain O2I value the classifier fails to generalize, and that threshold is not fixed - it shifts downward as training data grows, and the shift rate is task-specific (largest for the Majority and Topmost tasks, near zero for Multi). In low-data regimes, using a patch size smaller than the object (an object-to-patch ratio above 100%) raises validation accuracy by roughly 15 points on megapixel MNIST and 5 points on Swedish traffic signs relative to the IPS default, and attention maps show the smaller patch makes selection more decisive. Separately, when the Bezier noise thickness approaches the digit thickness, IPS stops converging and falls to random accuracy.
Load-bearing premise
In the main object-to-image ratio experiment, the number of noise components changes at the same time as the digit size through the formula noise digit size times -7.14 plus 1000, so the effect attributed to object size is entangled with noise density; the later noise-amount control uses 5000 training samples and a different thickness, so it does not rule out noise density as a confound in the low-data regime where the threshold effect is claimed.
Editorial extensions
If this is right
- If the central claim holds, IPS performance on small-ROI images cannot be summarized by a single object-to-image threshold; the threshold moves with the number of training samples and with the task, so benchmarks must report all three jointly.
- In low-data settings, practitioners should make the patch size equal to or smaller than the smallest expected region of interest, trading patch size for generalization rather than increasing labels.
- When noise or background structures resemble the target object in thickness, IPS convergence degrades sharply, so dataset realism must account for visual similarity, not just signal-to-noise ratio.
- Smaller patch sizes also reduce peak memory and per-batch runtime in the Swedish traffic signs experiments, so the generalization gain does not come at an efficiency cost.
- Attention maps show that lower O2I ratios make IPS assign informative attention to noise patches; the paper's patch-size finding implies that selecting patch scale below the ROI sharpens the attention contrast between object and noise.
Reading between the lines
- Beyond the paper, the O2I-by-dataset-size interaction predicts that other memory-efficient selectors, such as attention sampling or differentiable top-K, should show the same threshold shift if the failure is about patch-level salience rather than about IPS specifically.
- Beyond the paper, the thickness result suggests a testable conjecture: replacing the Bezier noise with simple straight lines of the same thickness should produce the same convergence collapse, since the paper's control experiments attribute the effect to thickness rather than to curve shape.
- Beyond the paper, the patch-size recommendation should be validated on datasets with a wide spread of object sizes, such as whole-slide histology images, where the smallest metastasis rather than the average object should determine patch scale.
- Beyond the paper, the fixed-canvas digital-scaling protocol could be reused as a diagnostic for label efficiency: the dataset size at which a model reaches a target accuracy at a given O2I ratio effectively measures how much training data is needed to compensate for object shrinkage.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies Iterative Patch Selection (IPS), a memory-efficient transformer-based patch classifier, under low object-to-image (O2I) ratios. It introduces a megapixel MNIST testbed with fixed canvas size, digit sizes varied from 28×28 to 112×112, and Bézier-curve noise. The main empirical claims are: (i) the O2I threshold below which IPS fails to generalize is affected by training dataset size, with the size of the interaction depending on the megapixel MNIST task; (ii) noise whose thickness resembles the digits causes IPS to fail to converge; and (iii) in low-data settings, tuning patch size smaller than the ROI improves validation accuracy by about +15% on megapixel MNIST and +5% on Swedish traffic signs. Code is released. The paper is empirical and does not derive its conclusions from fitted equations.
Significance. If the central claims hold, the paper provides useful evidence that the generalization behaviour previously observed on CNNs extends to iterative patch-selection transformers, and it identifies a simple, actionable intervention (patch-size tuning) for low-data, low-O2I regimes. The release of code and the use of multiple seeds in several experiments are strengths. However, the headline O2I/dataset-size interaction currently rests on a single experiment in which O2I is confounded with noise amount, and that experiment is reported without repeated seeds or error bars. The significance is therefore conditional on the confound being resolved and the quantitative claims being made reproducible.
major comments (3)
- [Section 4.1, noise formula] The central O2I experiment is confounded: digit size and noise count are co-varied through the stated relation noise = digit_size × (−7.14) + 1000. As written, this formula gives 800, 600, 400, and 200 noise components for digit sizes 28, 56, 84, and 112, while the text claims 800 for the lowest O2I and 4000 for the highest; either way, noise amount is not held constant across O2I levels. The control in Section 4.2 (Table 3) uses 5000 training samples and a fixed thickness of 1.925, so it does not rule out noise amount as the driving factor in the 800–4000 sample regime in which the dataset-size/O2I interaction is claimed. Please repeat the O2I sweep with a fixed noise count, or add a noise-count sweep at 1000 and 2000 training samples, and report those curves.
- [Section 4.1, Figure 4] The accuracy values in Section 4.1 and Figure 4 appear to be single runs without repeated seeds or error bars. Given that Tables 2 and 4 report standard deviations of 10–25 percentage points, single-run differences of 5–10 points (e.g., the 'Max' task at 2000 samples, 83% vs 79%) are not sufficient to establish the claimed interaction. Please report means and standard deviations over at least three seeds for each (O2I, dataset size, task) cell, or state explicitly that Figure 4 is already an average over seeds.
- [Abstract and Section 5, +15% claim] The claimed '+15% improvement for the megapixel MNIST' is not directly derivable from Table 4. Comparing the best small patch (25×25) with the O2P-equivalent original patch (150×150) gives per-task gains of 33, 22, 32, and 7 points, averaging about 24 points; comparing 50×50 with 150×150 gives gains of about 21, 9, 19, and 19 points, averaging about 17 points. Please specify exactly which patches and tasks are used to obtain the 15% figure. The Swedish traffic signs +5% (75×75 vs 100×100, Table 6) is clear and reproducible.
minor comments (5)
- [Section 4.2, random-accuracy calculation] The calculation for the 'Max' task contains an arithmetic error: C(8,2)/120 = 28/120 ≈ 0.233, not 0.30, so the listed probability for digit 9 should be 0.23. The observed random accuracy of about 29% therefore needs a different explanation, possibly arising from the three-identical-digit construction of the megapixel MNIST task.
- [Section 4.3.2] The subset-size sentence says '{25%, 50%, 100%} resulting in the following training set sizes {744, 372, 184}', but the order and value are inconsistent with Table 5, which shows training sizes 184, 372, and 747 for 25%, 50%, and 100% respectively. Please correct the values and their ordering.
- [Section 3, Equation 9] The Bézier formula is referred to as 'Equation 9' but the first numbered equation in the paper is Equation 1; the numbering is inconsistent, and the same formula appears inside Algorithm 1. Please unify the equation numbering and label the Bézier definition once.
- [General presentation] There are several typos and minor formatting issues, including 'disciminative' for 'discriminative' in Section 2.3, inconsistent use of 'O2P' percentages greater than 100% in Table 4 without explanation, and reference entries with inconsistent formatting. A final proofread would improve clarity.
- [Table 2] For thickness 2.2, the reported standard deviations of 0.0 for tasks 'Maj', 'Top', and 'Multi' across three seeds are surprising; please verify that these are not due to a reporting mistake, and if the value is correct, note that all seeds collapsed to the same degenerate solution.
Circularity Check
No circularity: the paper is an empirical benchmark study whose claims are measured outcomes, not quantities defined by their inputs.
full rationale
Scanning all sections, including the experimental setup, supplementary findings, and discussion, found no step in which a claimed prediction or first-principles result is defined in terms of its own inputs. The paper is an empirical study that builds a testbed (Bézier-curve noise, fixed canvas, varying digit size) and measures IPS validation accuracy. The central claim that the O2I threshold depends on training dataset size is an empirical generalization, not derived from an equation that embeds the conclusion. The authors' citations (Bergner et al. 2023; Pawlowski et al. 2020; Katharopoulos and Fleuret 2019) are external prior work, and there are no overlapping-author self-citations carrying the argument. The noise thickness of 1.925 is explicitly chosen post hoc as a calibration value after observing the thickness-2.0 collapse, but it is used as a fixed testbed condition, not fitted to a target outcome or renamed as a prediction. The patch-size recommendation in Section 4.3 is hyperparameter search over validation accuracy, not a quantity defined to equal the reported gain. The main validity concern, that in Section 4.1 the noise count varies with digit size via noise = digit_size*(-7.14) + 1000, is a potential confound in the experimental design and should be assessed as correctness risk, not circularity: it does not make the outcome equal to the input by construction. The paper is self-contained against external benchmarks and does not reduce any result to its own assumptions.
Assumptions & free parameters
free parameters (4)
- noise_count_formula_coefficients =
slope -7.14, intercept 1000
- noise_thickness =
1.925
- bezier_control_point_probabilities =
P(n=4)=3/9, P(n=6)=5/9, P(n=8)=1/9
- O2I_ratio_set =
{0.01, 0.034, 0.078, 0.13}%
assumptions (4)
- domain assumption The megapixel MNIST benchmark is a valid proxy for real-world high-resolution low-O2I classification tasks.
- domain assumption IPS implementation and hyperparameters from Bergner et al. (2023) are taken as given and are not re-validated.
- ad hoc to paper The noise generation with Bézier curves mimics adverse conditions found in real-world data, and the thickness of noise is the dominant factor over noise quantity.
- domain assumption Validation accuracy is a sufficient operationalization of 'generalization'; no formal threshold for 'fails to generalize' is defined.
Cite this review
Pith. "Pith review of On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification." pith.science (2026). https://pith.science/paper/XOERLVVN
@misc{pith2026241211237,
author = {Pith},
title = {Pith review of: On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/XOERLVVN}},
note = {Machine review of arXiv:2412.11237}
}
read the original abstract
Classifying large images with small or tiny regions of interest (ROI) is challenging due to computational and memory constraints. Weakly supervised memory-efficient patch selectors have achieved results comparable with strongly supervised methods. However, low signal-to-noise ratios and low entropy attention still cause overfitting. We explore these issues using a novel testbed on a memory-efficient cross-attention transformer with Iterative Patch Selection (IPS) as the patch selection module. Our testbed extends the megapixel MNIST benchmark to four smaller O2I (object-to-image) ratios ranging from 0.01% to 0.14% while keeping the canvas size fixed and introducing a noise generation component based on B\'ezier curves. Experimental results generalize the observations made on CNNs to IPS whereby the O2I threshold below which the classifier fails to generalize is affected by the training dataset size. We further observe that the magnitude of this interaction differs for each task of the Megapixel MNIST. For tasks "Maj" and "Top", the rate is at its highest, followed by tasks "Max" and "Multi" where in the latter, this rate is almost at 0. Moreover, results show that in a low data setting, tuning the patch size to be smaller relative to the ROI improves generalization, resulting in an improvement of + 15% for the megapixel MNIST and + 5% for the Swedish traffic signs dataset compared to the original object-to-patch ratios in IPS. Further outcomes indicate that the similarity between the thickness of the noise component and the digits in the megapixel MNIST gradually causes IPS to fail to generalize, contributing to previous suspicions.
Reference graph
Works this paper leans on
-
[1]
Abouelela, A., Abbas, H.M., Eldeeb, H., Wahdan, A.A., Nassar, S.M. (2005, July). Auto- mated vision system for localizing structural 13 defects in textile fabrics. Pattern Recogni- tion Letters, 26(10), 1435–1443, https:// doi.org/10.1016/j.patrec.2004.11.016
-
[2]
Ba, J., Mnih, V., Kavukcuoglu, K. (2015, April). Multiple Object Recognition with Visual Attention. arXiv. Retrieved 2024- 07-05, from http://arxiv.org/abs/1412.7755 (arXiv:1412.7755 [cs])
arXiv 2015
-
[3]
Baydas, S., & Karakas, B. (2019, Decem- ber). Defining a curve as a Bezier curve. Journal of Taibah Univer- sity for Science , 13(1), 522–528, https://doi.org/10.1080/16583655.2019 .1601913 Retrieved 2024-07-23, from https://doi.org/10.1080/16583655.2019.1601913 (Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/16583655.2019.1601913)
-
[4]
Iterative Patch Selection for High-Resolution Image Recognition
Bergner, B., Lippert, C., Mahendran, A. (2023, March). Iterative Patch Selection for High-Resolution Image Recogni- tion. arXiv. Retrieved 2024-05-29, from http://arxiv.org/abs/2210.13007 (arXiv:2210.13007 [cs, eess])
work page Pith review arXiv 2023
-
[5]
Blondel, M., Teboul, O., Berthet, Q., Djolonga, J. (2020, November). Fast Differen- tiable Sorting and Ranking. Proceedings of the 37th International Conference on Machine Learning (pp. 950–959). PMLR. Retrieved 2024-06-01, from https://proceedings.mlr.press/v119/blondel20a.html (ISSN: 2640-3498)
work page 2020
-
[6]
Unterthiner, T. (2021). Differentiable Patch Selection for Image Recognition. (pp. 2351– 2360)
work page 2021
-
[7]
Lavergne, A., Courtiol, P. (2020, Decem- ber). Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology. arXiv. Retrieved 2024-05- 30, from http://arxiv.org/abs/2012.03583 (arXiv:2012.03583 [cs, eess])
arXiv 2020
-
[8]
Li, K., Fei-Fei, L. (2009, June). Ima- geNet: A large-scale hierarchical image database. 2009 IEEE Conference on Com- puter Vision and Pattern Recognition(pp. 248–255). Retrieved 2024-07-17, from https://ieeexplore.ieee.org/document/5206848 (ISSN: 1063-6919)
arXiv 2009
Show all 25 references
-
[9]
(2024, March)
Gadermayr, M., & Tschuchnig, M. (2024, March). Multiple instance learning for digital pathology: A review of the state- of-the-art, limitations & future potential. Computerized Medical Imaging and Graph- ics, 112, 102337, https://doi.org/10.1016/ j.compmedimag.2024.102337
2024
-
[10]
(2015, Decem- ber)
He, K., Zhang, X., Ren, S., Sun, J. (2015, Decem- ber). Deep Residual Learning for Image Recognition. arXiv. Retrieved 2024-07- 17, from http://arxiv.org/abs/1512.03385 (arXiv:1512.03385 [cs])
2015 arXiv
-
[11]
(2018, June)
Ilse, M., Tomczak, J.M., Welling, M. (2018, June). Attention-based Deep Multiple Instance Learning. arXiv. Retrieved 2024-06- 01, from http://arxiv.org/abs/1802.04712 (arXiv:1802.04712 [cs, stat])
2018 arXiv
-
[12]
(2019, July)
Katharopoulos, A., & Fleuret, F. (2019, July). Processing Megapixel Images with Deep Attention-Sampling Mod- els. arXiv. Retrieved 2024-05-31, from http://arxiv.org/abs/1905.03711 (arXiv:1905.03711 [cs, stat])
2019 arXiv
-
[13]
(2021, Decem- ber)
Kong, F., & Henao, R. (2021, Decem- ber). Efficient Classification of Very Large Images with Tiny Objects. arXiv. Retrieved 2024-05-29, from http://arxiv.org/abs/2106.02694 (arXiv:2106.02694 [cs]) Krizhevsky,A.,Sutskever,I.,Hinton,G.E. (2012). ImageNet classification with deep...
2012 arXiv
-
[14]
(2018, June)
LaLonde, R., Zhang, D., Shah, M. (2018, June). ClusterNet: Detecting Small Objects in Large Scenes by Exploit- ing Spatio-Temporal Information. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4003–4012). Salt Lake City, UT, USA: IEEE. Retrieved 2024-06...
2018
-
[15]
Larsson, F., & Felsberg, M. (2011). Using Fourier Descriptors and Spatial Models for Traffic Sign Recognition. A. Heyden & F. Kahl (Eds.), Image Analysis (pp. 238– 249). Berlin, Heidelberg: Springer
2011
-
[16]
(2014, June)
Mnih, V., Heess, N., Graves, A., Kavukcuoglu, K. (2014, June). Recurrent Models of Visual Attention. arXiv. Retrieved 2024- 06-22, from http://arxiv.org/abs/1406.6247 (arXiv:1406.6247 [cs, stat])
2014 arXiv
-
[17]
Joost, S
Ofli, F., Meier, P., Imran, M., Castillo, C., Tuia, D., Rey, N., ... Joost, S. (2016, March). CombiningHumanComputingandMachine Learning to Make Sense of Big (Aerial) Data for Disaster Response. Big Data, 4(1), 47– 59, https://doi.org/10.1089/big.2014.0064 (Publisher: Mary Ann...
2016
-
[18]
(2020, Jan- uary)
Pawlowski, N., Bhooshan, S., Ballas, N., Ciompi, F., Glocker, B., Drozdzal, M. (2020, Jan- uary). Needles in Haystacks: On Classifying Tiny Objects in Large Images.arXiv
2020
-
[19]
(2013, October)
Qin, X., Hu, G., Zhang, N., Shen, X., Yang, Y. (2013, October). A novel extension to the polynomial basis functions describing Bezier curves and surfaces of degreen with multi- ple shape parameters.Applied Mathematics and Computation, 223, 1–16, https://doi .org/10.1016/j.amc....
2013 doi
-
[20]
(2021, October)
Ji, X., Zhang, Y. (2021, October). Trans- MIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification. arXiv. Retrieved 2024-06- 24, from http://arxiv.org/abs/2106.00908 (arXiv:2106.00908 [cs])
2021 arXiv
-
[21]
Thandiackal, K., Chen, B., Pati, P., Jaume, G., Williamson, D.F.K., Gabrani, M., Gok- sel, O. (2022). Differentiable Zooming for Multiple Instance Learning on Whole- Slide Images. S. Avidan, G. Brostow, M. Cissé, G.M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022...
2022
-
[22]
Polo- sukhin, I
Vaswani, A., Shazeer, N., Parmar, N., Uszkor- eit, J., Jones, L., Gomez, A.N., ... Polo- sukhin, I. (2023, August). Attention Is All You Need. arXiv. Retrieved 2024-07- 22, from http://arxiv.org/abs/1706.03762 (arXiv:1706.03762 [cs])
2023 arXiv
-
[23]
(2019, Decem- ber)
Xue, L., Li, X., Zhang, N.L. (2019, Decem- ber). Not All Attention Is Needed: Gated Attention Network for Sequence Data. arXiv. Retrieved 2024-08- 12, from http://arxiv.org/abs/1912.00349 (arXiv:1912.00349 [cs, stat])
2019 arXiv
-
[24]
(2017, February)
Vinyals, O. (2017, February). Under- standing deep learning requires rethinking generalization. arXiv
2017
-
[25]
(2024, April)
Zhang, Y., Li, H., Sun, Y., Zheng, S., Zhu, C., Yang, L. (2024, April). Attention- Challenging Multiple Instance Learning for Whole Slide Image Classification. arXiv. Retrieved 2024-06-17, from http://arxiv.org/abs/2311.07125 15
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.