{"id":"6903ff27-d42b-46e2-9492-d229ef2d2751","arxiv_id":"2412.11237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The generalization threshold for Iterative Patch Selection on low object-to-image-ratio images depends on training set size and task, and smaller patches relative to the object improve low-data accuracy.","lead":"This paper tests how a memory-efficient patch-selection classifier, IPS, behaves on huge images where the object of interest is tiny, and finds that the amount of training data needed depends on both object size and the specific classification task. It also shows that using patches smaller than the object can improve accuracy in low-data settings on two benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main O2I experiment varies noise count together with digit size, so the central dataset-size/O2I interaction claim may reflect noise density rather than object-to-image ratio.","rationale":"The reader's verdict is CONDITIONAL and the reader's weakest_assumption is exactly the noise-density confound in Section 4.1. My independent reading confirms this is the single most load-bearing issue: the central claim states that the O2I threshold below which IPS fails to generalize is affected by training dataset size, and the only experiment supporting that claim changes O2I and noise count simultaneously. The paper itself gives evidence that noise properties matter greatly (Table 2), and its own Table 3 control is run only at 5000 samples, not in the low-data regime of Figure 4. The concern does not, however, force rejection: the paper is an empirical study, the confound is explicitly described in the text, and a relatively straightforward control experiment would settle it. The patch-size findings (Section 4.3) are less directly affected because they compare patch sizes at a fixed digit size and fixed noise setup. I agree with the reader that the verdict should remain CONDITIONAL: the realignment of O2I and noise density is addressable and may leave the qualitative conclusions intact, but the central claim cannot be accepted without the control.","tokens_in":16114,"tokens_out":1571,"duration_ms":12221,"concrete_test":"Run the Section 4.1 experiment again on the 3000x3000 canvas while holding the number and thickness of noise components fixed across all four digit sizes (e.g., fix noise at 800 curves with thickness 1.925, or a number matched to the smallest digit size). If the validation-accuracy-versus-dataset-size curves no longer show a systematic ordering by O2I ratio, then the observed interaction in Figure 4 was driven by noise density and the central claim fails. If the ordering persists with constant noise, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim, that the O2I threshold below which IPS fails to generalize is affected by training dataset size, rests on the experiments in Section 4.1. However, in that section the digit size is varied while the noise count is changed according to noise = digit_size*(-7.14) + 1000. Thus O2I is not varied independently: the smallest digits (28x28, O2I 0.01%) are paired with 800 noise components and the largest digits (112x112, O2I 0.14%) with 4000 noise components. A lower O2I therefore also means a lower noise density, and the improved accuracy at higher O2I could be caused by fewer distractors rather than by the ratio of object size to image size. The authors explicitly acknowledge this confound in Section 4.2: Table 3 shows that noise amount has little effect when data is abundant (5000 samples) and thickness is held at 1.925, but that control is not carried out in the low-data regime (4000, 2000, 1000, 800 samples) where the threshold interaction is claimed. Given that Table 2 shows that small differences in noise digit thickness cause abrupt performance collapses (e.g., Maj drops from 58.4% to 10.2% between thickness 1.8 and 2.0), the dataset-size-dependent patterns in Figure 4 could plausibly be driven by changes in noise density rather than by O2I per se. This is the most load-bearing concern because it directly undermines the paper's headline generalization claim, and it is empirically testable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Iterative Patch Selection (IPS), a memory-efficient transformer-based patch classifier, under low object-to-image (O2I) ratios. It introduces a megapixel MNIST testbed with fixed canvas size, digit sizes varied from 28×28 to 112×112, and Bézier-curve noise. The main empirical claims are: (i) the O2I threshold below which IPS fails to generalize is affected by training dataset size, with the size of the interaction depending on the megapixel MNIST task; (ii) noise whose thickness resembles the digits causes IPS to fail to converge; and (iii) in low-data settings, tuning patch size smaller than the ROI improves validation accuracy by about +15% on megapixel MNIST and +5% on Swedish traffic signs. Code is released. The paper is empirical and does not derive its conclusions from fitted equations.","tokens_in":16481,"tokens_out":6980,"duration_ms":60758,"significance":"If the central claims hold, the paper provides useful evidence that the generalization behaviour previously observed on CNNs extends to iterative patch-selection transformers, and it identifies a simple, actionable intervention (patch-size tuning) for low-data, low-O2I regimes. The release of code and the use of multiple seeds in several experiments are strengths. However, the headline O2I/dataset-size interaction currently rests on a single experiment in which O2I is confounded with noise amount, and that experiment is reported without repeated seeds or error bars. The significance is therefore conditional on the confound being resolved and the quantitative claims being made reproducible.","major_comments":[{"comment":"The central O2I experiment is confounded: digit size and noise count are co-varied through the stated relation noise = digit_size × (−7.14) + 1000. As written, this formula gives 800, 600, 400, and 200 noise components for digit sizes 28, 56, 84, and 112, while the text claims 800 for the lowest O2I and 4000 for the highest; either way, noise amount is not held constant across O2I levels. The control in Section 4.2 (Table 3) uses 5000 training samples and a fixed thickness of 1.925, so it does not rule out noise amount as the driving factor in the 800–4000 sample regime in which the dataset-size/O2I interaction is claimed. Please repeat the O2I sweep with a fixed noise count, or add a noise-count sweep at 1000 and 2000 training samples, and report those curves.","section":"Section 4.1, noise formula"},{"comment":"The accuracy values in Section 4.1 and Figure 4 appear to be single runs without repeated seeds or error bars. Given that Tables 2 and 4 report standard deviations of 10–25 percentage points, single-run differences of 5–10 points (e.g., the 'Max' task at 2000 samples, 83% vs 79%) are not sufficient to establish the claimed interaction. Please report means and standard deviations over at least three seeds for each (O2I, dataset size, task) cell, or state explicitly that Figure 4 is already an average over seeds.","section":"Section 4.1, Figure 4"},{"comment":"The claimed '+15% improvement for the megapixel MNIST' is not directly derivable from Table 4. Comparing the best small patch (25×25) with the O2P-equivalent original patch (150×150) gives per-task gains of 33, 22, 32, and 7 points, averaging about 24 points; comparing 50×50 with 150×150 gives gains of about 21, 9, 19, and 19 points, averaging about 17 points. Please specify exactly which patches and tasks are used to obtain the 15% figure. The Swedish traffic signs +5% (75×75 vs 100×100, Table 6) is clear and reproducible.","section":"Abstract and Section 5, +15% claim"}],"minor_comments":[{"comment":"The calculation for the 'Max' task contains an arithmetic error: C(8,2)/120 = 28/120 ≈ 0.233, not 0.30, so the listed probability for digit 9 should be 0.23. The observed random accuracy of about 29% therefore needs a different explanation, possibly arising from the three-identical-digit construction of the megapixel MNIST task.","section":"Section 4.2, random-accuracy calculation"},{"comment":"The subset-size sentence says '{25%, 50%, 100%} resulting in the following training set sizes {744, 372, 184}', but the order and value are inconsistent with Table 5, which shows training sizes 184, 372, and 747 for 25%, 50%, and 100% respectively. Please correct the values and their ordering.","section":"Section 4.3.2"},{"comment":"The Bézier formula is referred to as 'Equation 9' but the first numbered equation in the paper is Equation 1; the numbering is inconsistent, and the same formula appears inside Algorithm 1. Please unify the equation numbering and label the Bézier definition once.","section":"Section 3, Equation 9"},{"comment":"There are several typos and minor formatting issues, including 'disciminative' for 'discriminative' in Section 2.3, inconsistent use of 'O2P' percentages greater than 100% in Table 4 without explanation, and reference entries with inconsistent formatting. A final proofread would improve clarity.","section":"General presentation"},{"comment":"For thickness 2.2, the reported standard deviations of 0.0 for tasks 'Maj', 'Top', and 'Multi' across three seeds are surprising; please verify that these are not due to a reporting mistake, and if the value is correct, note that all seeds collapsed to the same degenerate solution.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the patch-size finding is practically useful. The main risk is that the headline generalization claim is currently supported only by a confounded and non-repeated experiment. I would encourage the editor to request the fixed-noise and multi-seed reruns before considering the paper further, rather than treating the current version as ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper with a solid patch-size finding, but the headline O2I threshold claim sits on an experiment where object size and noise count move together. In Section 4.1, noise count is set by noise = digit_size*(-7.14)+1000. The paper states this gives 800 noise components at the lowest O2I ratio and 4000 at the highest, but the formula gives 800 at 28x28 and about 200 at 112x112. Either way, O2I and noise count are entangled: higher O2I coincides with fewer distractors (or many more, depending on which line you trust), so you cannot attribute the accuracy differences to object-to-image ratio alone. The stress-test note gets the direction wrong, but the core concern stands. The later noise-amount control at 5000 samples does not cover the low-data regime (800-4000 samples) where the threshold interaction is claimed, so it does not resolve the confound. Also, the O2I experiment has no error bars and no repeated seeds, and the abstract's +15% gain for megapixel MNIST is not directly derivable from Table 4; the gains vary by task and baseline. What the paper does well: the fixed-canvas scaling is a clean methodological alternative to prior canvas-scaling work, the Bezier-curve noise generator is a concrete new testbed, and the patch-size experiments on Swedish traffic signs are convincing: 75x75 beats 100x100 by about 5 points, with lower memory, and the attention maps illustrate the mechanism. The paper is also transparent about failed regularizers. This is a paper for people working on memory-efficient patch classifiers and tiny-ROI testbeds. It deserves a serious referee: the testbed and patch-size guidance are worth keeping, and the central claim is fixable with a cleaner experiment that varies O2I with noise fixed and separately varies noise count across the low-data sample sizes. I'd send it to review rather than desk reject.","headline":"Useful empirical study with a real confound in the central O2I experiment; the patch-size findings and testbed are worth keeping, but the headline claim needs a cleaner experiment.","tokens_in":610,"tokens_out":3640,"would_cite":true,"duration_ms":59771,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Iterative Patch Selection fails to generalize below an object-to-image threshold that depends on training data size and task, and that in low-data settings a patch size smaller than the region of interest recovers…","keywords":["high-resolution image classification","Iterative Patch Selection","object-to-image ratio","megapixel MNIST","patch size","overfitting","weakly supervised learning","Bezier curve noise"],"falsifier":"Run the same 3000 by 3000 megapixel MNIST setup with a fixed number of noise components, say 800, across digit sizes 28, 56, 84, and 112 while keeping noise thickness fixed; if the accuracy-versus-dataset-size pattern disappears, the reported O2I threshold is a noise-density effect rather than an object-size effect.","tokens_in":15918,"feed_emoji":"🧩","tokens_out":4609,"duration_ms":40200,"temperature":0.7,"pith_summary":"The paper tries to establish whether Iterative Patch Selection, a memory-efficient patch-based classifier, breaks down on high-resolution images with very small objects, and what can be done about it. It extends the megapixel MNIST benchmark to a fixed canvas with four object-to-image ratios and replaces linear noise with Bezier curves that resemble digits. Experiments show the object-to-image ratio below which IPS stops generalizing moves with the training set size, and the strength of that interaction depends on the task. In low-data settings, a patch size smaller than the region of interest improves validation accuracy by about 15 points on megapixel MNIST and 5 points on Swedish traffic signs. The paper also finds that when noise thickness approaches digit thickness, IPS stops converging entirely.","feed_headline":"Tiny objects need either more data or smaller patches","feed_subtitle":"Patch selection fails below an object-to-image threshold that data size shifts; smaller patches recover 15% accuracy.","key_machinery":"The central mechanism is Iterative Patch Selection: the image is split into patches, batches are encoded and scored by a no-gradient cross-attention module, only the top M most salient patches are kept in memory after each iteration, and the survivors are aggregated by an attention-weighted mean. The testbed couples this module with a fixed-canvas megapixel MNIST variant whose object-to-image ratio changes by scaling digit size, and a Bezier-curve noise generator whose control-point counts and line thickness mimic digits. Patch size, training set size, and O2I ratio are the controlled variables that expose the generalization boundary.","core_discovery":"Iterative Patch Selection, a weakly supervised transformer-based classifier that keeps only the top M salient patches, has a generalization threshold in object-to-image ratio: below a certain O2I value the classifier fails to generalize, and that threshold is not fixed - it shifts downward as training data grows, and the shift rate is task-specific (largest for the Majority and Topmost tasks, near zero for Multi). In low-data regimes, using a patch size smaller than the object (an object-to-patch ratio above 100%) raises validation accuracy by roughly 15 points on megapixel MNIST and 5 points on Swedish traffic signs relative to the IPS default, and attention maps show the smaller patch makes selection more decisive. Separately, when the Bezier noise thickness approaches the digit thickness, IPS stops converging and falls to random accuracy.","pith_inferences":["Beyond the paper, the O2I-by-dataset-size interaction predicts that other memory-efficient selectors, such as attention sampling or differentiable top-K, should show the same threshold shift if the failure is about patch-level salience rather than about IPS specifically.","Beyond the paper, the thickness result suggests a testable conjecture: replacing the Bezier noise with simple straight lines of the same thickness should produce the same convergence collapse, since the paper's control experiments attribute the effect to thickness rather than to curve shape.","Beyond the paper, the patch-size recommendation should be validated on datasets with a wide spread of object sizes, such as whole-slide histology images, where the smallest metastasis rather than the average object should determine patch scale.","Beyond the paper, the fixed-canvas digital-scaling protocol could be reused as a diagnostic for label efficiency: the dataset size at which a model reaches a target accuracy at a given O2I ratio effectively measures how much training data is needed to compensate for object shrinkage."],"forward_implications":["If the central claim holds, IPS performance on small-ROI images cannot be summarized by a single object-to-image threshold; the threshold moves with the number of training samples and with the task, so benchmarks must report all three jointly.","In low-data settings, practitioners should make the patch size equal to or smaller than the smallest expected region of interest, trading patch size for generalization rather than increasing labels.","When noise or background structures resemble the target object in thickness, IPS convergence degrades sharply, so dataset realism must account for visual similarity, not just signal-to-noise ratio.","Smaller patch sizes also reduce peak memory and per-batch runtime in the Swedish traffic signs experiments, so the generalization gain does not come at an efficiency cost.","Attention maps show that lower O2I ratios make IPS assign informative attention to noise patches; the paper's patch-size finding implies that selecting patch scale below the ROI sharpens the attention contrast between object and noise."],"supporting_citations":[{"why":"Introduces Iterative Patch Selection itself, the baseline setup, the original megapixel MNIST patch settings, and the state-of-the-art results the paper extends and tests.","marker":"Bergner et al., 2023"},{"why":"Provides the Needle MNIST benchmark and the prior finding that CNN generalization depends on dataset size and O2I threshold, which this paper generalizes to IPS.","marker":"Pawlowski et al., 2020"},{"why":"Introduces the megapixel MNIST dataset and attention-sampling patch selection, including the linear noise component and task setup adapted here.","marker":"Katharopoulos & Fleuret, 2019"},{"why":"Supplies the differentiable Top-K patch selection baseline that IPS is compared against as a memory-efficient weakly supervised method.","marker":"Cordonnier et al., 2021"},{"why":"Provides evidence that patch selectors overlook very small regions of interest due to low attention, motivating the failure mode studied in this paper.","marker":"Thandiackal et al., 2022"},{"why":"Is the source of the Swedish traffic signs dataset used to validate the patch-size recommendation outside MNIST.","marker":"Larsson & Felsberg, 2011"}],"fun_headline_variants":["Patch size beats data when objects are tiny","Smaller patches recover 15% accuracy in low-data settings","IPS fails below a data-dependent object-to-image threshold","Tiny ROIs? Shrink patches to boost accuracy 15 points","Bezier noise thickness can break patch selection generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In the main object-to-image ratio experiment, the number of noise components changes at the same time as the digit size through the formula noise digit size times -7.14 plus 1000, so the effect attributed to object size is entangled with noise density; the later noise-amount control uses 5000 training samples and a different thickness, so it does not rule out noise density as a confound in the low-data regime where the threshold effect is claimed.","fun_headline_variants_meta":{"raw":{"variants":["Patch size beats data when objects are tiny","Smaller patches recover 15% accuracy in low-data settings","IPS fails below a data-dependent object-to-image threshold","Tiny ROIs? Shrink patches to boost accuracy 15 points","Bezier noise thickness can break patch selection generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1615,"prompt_tokens":1018,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":634,"tokens_out":597,"duration_ms":5885,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:09:14.144163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 3000 by 3000 megapixel MNIST setup with a fixed number of noise components, say 800, across digit sizes 28, 56, 84, and 112 while keeping noise thickness fixed; if the accuracy-versus-dataset-size pattern disappears, the reported O2I threshold is a noise-density effect rather than an object-size effect.","supporting_citations":[{"cited_title":"Iterative Patch Selection for High-Resolution Image Recognition","cited_arxiv_id":"2210.13007","evidence_quote":"Introduces Iterative Patch Selection itself, the baseline setup, the original megapixel MNIST patch settings, and the state-of-the-art results the paper extends and tests."},{"cited_title":"(2020, Jan- uary)","cited_arxiv_id":null,"evidence_quote":"Provides the Needle MNIST benchmark and the prior finding that CNN generalization depends on dataset size and O2I threshold, which this paper generalizes to IPS."},{"cited_title":"Processing Megapixel Images with Deep Attention-Sampling Models","cited_arxiv_id":"1905.03711","evidence_quote":"Introduces the megapixel MNIST dataset and attention-sampling patch selection, including the linear noise component and task setup adapted here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that patch selectors overlook very small regions of interest due to low attention, motivating the failure mode studied in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the source of the Swedish traffic signs dataset used to validate the patch-size recommendation outside MNIST."}],"review_version":1}