{"id":"838b684f-f99e-424c-bd01-6cd05d93a473","arxiv_id":"1908.06037","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Standard convolutional networks can classify tiny objects in large images only above a certain object-to-image ratio, and the training data needed to reach that point rises rapidly as the object gets smaller.","lead":"The authors built two controlled testbeds, one from MNIST digits and one from histopathology images, to study image classification when the object that determines the label is tiny within a large image. They found a sharp object-size limit below which standard convolutional networks fail to generalize, and that the amount of training data needed grows quickly as the object shrinks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clutter count, not O2I, may drive the nMNIST scaling law; the generator varies both together.","rationale":"The central claim is an empirical scaling law, so the controlled variable must be O2I alone. The nMNIST generation protocol violates this by co-varying distractor count with canvas size, making the law unfalsifiable within the current data: any monotone increase in required data across the five O2I rows is equally consistent with \"smaller object\" or \"more distractors.\" The proposed test separates these hypotheses. If the law fails under fixed clutter count, the abstract's wording should be weakened to constant-clutter-density conditions or restated as a distractor-count effect rather than a general property of O2I. The reader's conditional verdict already targets this confound, so I recommend no change in verdict; the additional experiment would turn the condition into a decisive check. The paper's self-reported limitations — small nCAMELYON and reliance on synthetic nMNIST — increase rather than decrease the need for this ablation, since no real-data dataset-size sweep supports the scaling law.","tokens_in":18598,"tokens_out":6281,"duration_ms":63496,"concrete_test":"Regenerate nMNIST at O2I ∈ {1.2, 0.3, 0.075}% using the same 256/512/1024 canvases but fix the clutter count at 25 (the 256×256 value) instead of 25/100/400; repeat the max-pooling RF=33 ResNet-50 experiments for training sizes {1400, 2819, 5638, 7500, 11276, 22552} and rebuild Fig. 6b. If the minimum-samples-to-85% curve still rises steeply as O2I falls, the O2I explanation survives; if it flattens, the original inverse-O2I scaling is an artifact of the simultaneously increasing distractor count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline law — that training-set size scales rapidly with inverse object-to-image ratio — is read off nMNIST in Section 3.1 (Fig. 6b), but in Section 2.1 O2I is not varied independently. To keep \"approximately constant clutter density\", the generator pairs each O2I value with a different number of distractors: 19.1%→2, 4.8%→5, 1.2%→25, 0.3%→100, and 0.075%→400 clutter digits. The low-O2I condition is therefore also the 200x-more-distractors condition, and the positive/negative decision becomes \"find one 3 among 401 digits\" rather than \"find a 28x28 3 in a 1024x1024 image.\" Because no dataset-size sweep is run on nCAMELYON, there is no independent evidence for the inverse-O2I scaling outside this confounded generator. The paper's own limitations note (small nCAMELYON, synthetic nMNIST) does not remove this within-generator confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two synthetic 'needle-in-a-haystack' testbeds, nMNIST and nCAMELYON, to study image-level binary classification when the object of interest occupies a very small fraction of the image (low object-to-image, O2I, ratio). Using a BagNet-style backbone with controlled receptive fields, capacities, and four global pooling operations, the authors report four empirical observations: (1) there is an O2I limit below which CNNs fail to generalize, and the amount of training data needed for generalization scales rapidly with the inverse of O2I; (2) higher-capacity models generalize better; (3) matching the receptive field to the object size is beneficial on nMNIST but not on nCAMELYON; and (4) global pooling choice affects optimization at the smallest O2I values. The main evidence for the headline scaling law comes from Fig. 6b, which is computed from nMNIST experiments over six training-set sizes and six seeds per configuration.","tokens_in":18761,"tokens_out":4477,"duration_ms":44254,"significance":"The paper addresses a practically important and understudied regime, and the testbed itself is a useful resource: nMNIST provides controlled O2I variation, and the experiments are unusually extensive for a study of this type, with 48 architectures, six seeds per run, and more than 750 reported experiments. If the scaling claim holds, it quantifies a real bottleneck for image-level CNN classifiers on tiny-object tasks. The code-release statement, if honored, would make the testbed reproducible. However, the central quantitative claim about inverse-O2I scaling is currently entangled with a second, uncontrolled factor, distractor count, and the nCAMELYON dataset is too small and too coarsely controlled to provide independent confirmation. The paper's usefulness as a testbed is therefore stronger than the evidence for its headline law.","major_comments":[{"comment":"The nMNIST generator does not vary O2I independently of clutter count. To maintain approximately constant clutter density, the canvas sizes 64, 128, 256, 512, and 1024 pixels are paired with 2, 5, 25, 100, and 400 clutter digits, respectively. Every low-O2I condition is therefore also a high-distractor condition, and the positive decision at 0.075% O2I is 'find one 3 among 401 digits' rather than 'find a 28x28 object in a 1024x1024 image'. The abstract's claim that required training-set size scales rapidly with inverse O2I is read directly from Fig. 6b, which uses only this confounded sweep; no dataset-size sweep is run on nCAMELYON. Please add control experiments that vary O2I at a fixed clutter count (e.g., keep 25 distractors while enlarging the canvas), or vary clutter count at fixed O2I, and report whether the scaling law still holds when the two factors are separated.","section":"Section 2.1 and Section 3.1 (Fig. 6b)"},{"comment":"The reported validation and test accuracies in Figs. 6a, 7, and 8 condition on optimization success: Fig. 6a averages only pipeline configurations that surpassed 70% training accuracy, and Figs. 7 and 8 report only runs that fit the training data, replacing non-fitting runs with random accuracy and a textured bar. This conditioning is stated in the captions, but it means the heatmaps and bar heights reflect generalization conditional on successful training, not the unconditional behavior of the architectures. Please report the fraction of runs that satisfy the conditioning, and show at least one unconditional accuracy result (or state explicitly whether the qualitative conclusions change when all runs are included).","section":"Section 3.1 (Figs. 6, 7, 8)"}],"minor_comments":[{"comment":"The abbreviation 'nMINIST' appears instead of 'nMNIST'; please fix this typo.","section":"Section 3.1 (Inductive bias paragraph)"},{"comment":"The sentence 'We maintain approximately constant clutter density over different O2I ratios' would be easier to verify if the actual densities (clutter count multiplied by digit area divided by canvas area) were listed for each O2I value, since the chosen counts give only approximate constancy.","section":"Section 2.1"},{"comment":"Please add explicit axis labels and a description of how 'minimum required training set size' is obtained from the six tested sizes; currently the reader must infer whether the values are interpolated or chosen among the grid points.","section":"Figure 6b"},{"comment":"The center-wise train/validation/test split is described only in the supplementary material; it should be stated in the main text because it is important for interpreting the nCAMELYON results.","section":"Section 2.1 (nCAMELYON)"},{"comment":"There are small typographical issues, e.g., 'differen O2I ratios' in Table 1 and inconsistent use of 'O2IR' versus 'O2I ratio' in the supplementary text; please standardize.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a testbed contribution with a plausible but not yet established quantitative law. The nMNIST confound between O2I and clutter count is the main risk to the headline claim; if the authors add even a small set of control experiments that separates these factors, the paper could become a solid empirical study. The conditioning of Figures 6-8 on training success should also be made transparent in the main text. No issues with novelty disclosure or citation patterns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"To give you the short version: this paper is a solid empirical contribution that deserves a serious referee, but the headline scaling law is less clean than it looks. The testbed—nMNIST plus nCAMELYON—and the 48-architecture sweep are genuinely useful. Six seeds per run and 750+ experiments is real effort. The qualitative findings are plausible: higher capacity helps, receptive-field matching helps, pooling matters at extreme low signal. And the paper is honest about the main limitations (synthetic data, small real dataset). I'd absolutely send it to review rather than desk reject.\n\nThe soft spot is the inverse-O2I scaling claim. To keep clutter density constant, the nMNIST generator pairs each O2I level with a different number of distractors: 2 at 19.1%, 5 at 4.8%, 25 at 1.2%, 100 at 0.3%, 400 at 0.075%. So the low-O2I condition is also the 200x-more-distractors condition. Figure 6b, which is the core evidence for \"data required scales rapidly with inverse O2I,\" cannot separate O2I from clutter count. The abstract's phrasing gives the O2I ratio the causal role; the setup doesn't support that cleanly. That's not a reason to reject, but it is a reason to ask for a decoupled ablation—vary object size on a fixed canvas, or vary clutter count at a fixed O2I—before calling it a scaling law. It could be that the real driver is the number of distractors, which would still be interesting but is a different statement.\n\nTwo smaller points. Figures 6a and 7/8 condition on runs that crossed a training-accuracy threshold; that's disclosed, but it means the reported numbers are optimistic relative to \"all seeds.\" On nCAMELYON, the unique-lesion counts are small (20–235 per config), so the pool of real signal is limited; a center-bias or texture baseline would strengthen the claim that the model is looking at lesions rather than site-specific stain.\n\nBottom line: worth a round of revisions, not worth a knock-out. If you work on medical or hyperspectral classification, cite it for the testbed, but read Figure 6 with a skeptical eye.","headline":"Useful testbed and a lot of honest empiricism, but the headline O2I scaling law is confounded by clutter count in nMNIST.","tokens_in":19311,"tokens_out":3180,"would_cite":true,"duration_ms":28434,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that image-level CNN classifiers have a minimum object size for a given training budget, and that the training data needed rises steeply as the object-to-image ratio shrinks.","keywords":["tiny object classification","object-to-image ratio","signal-to-noise ratio","CNN generalization","dataset scaling","global pooling","histopathology","receptive field"],"falsifier":"Re-run the 1024-by-1024 digit experiment with the same 0.075% object-to-image ratio but with, say, 10 clutter digits instead of 400; if the network then classifies above chance, the O2I limit is really a clutter-count limit, and the paper's headline scaling would not transfer to constant-clutter settings.","tokens_in":18399,"feed_emoji":"🔍","tokens_out":9856,"duration_ms":87215,"temperature":0.7,"pith_summary":"This paper tries to establish that convolutional neural networks trained only with image-level labels have a hard lower limit on how small the class-defining object can be relative to the whole image, and that this limit moves when the training set changes. On a controlled testbed that varies the object-to-image ratio from roughly 19% down to 0.075%, networks generalize well at larger ratios and then fall to chance at a threshold; the smaller the object, the more training data is needed, and the required amount grows rapidly as the ratio shrinks. The paper argues this behavior appears in both a simple digit-classification task and a histopathology task, so it is not an artifact of one domain. If the claim holds, it matters for medical and remote-sensing applications where the signal can occupy a tiny fraction of the image and pixel-level labels are costly.","feed_headline":"Tiny objects defeat neural nets until datasets grow sharply","feed_subtitle":"Training data for image-level classification must rise steeply as the target shrinks.","key_machinery":"The central object is the object-to-image (O2I) ratio, defined as the area of the class-relevant object divided by the area of the image. The argument is carried by two controlled datasets: a synthetic cluttered-digit set the paper calls nMNIST, which places a fixed 28-by-28 digit on canvases from 64-by-64 to 1024-by-1024 pixels while scaling the number of distractor digits from 2 to 400 to keep clutter density roughly constant, and a histopathology crop set the paper calls nCAMELYON, derived from pixel-level lesion annotations. Around this testbed the paper varies three network properties independently: receptive field size, model capacity, and the global pooling operation, using a convolutional backbone that permits explicit receptive-field control.","core_discovery":"On its own terms, the paper's central claim is that there is a minimum object-to-image ratio for a given CNN training budget, and that below this ratio image-level classifiers do not beat random guessing on held-out data. More data lowers the floor, but only at a steep cost: the number of training examples needed to reach a fixed validation accuracy rises quickly as the object-to-image ratio drops, so the gains from simply collecting more data diminish sharply. The paper also reports that at small object-to-image ratios higher-capacity models generalize better, that a receptive field matched to or larger than the object helps when the object size is known, and that global max pooling is the most reliable choice for very small signals, while pooling choice matters little at larger ratios.","pith_inferences":["Because the digit testbed raises canvas size and clutter count together, the inverse-O2I scaling may actually measure how well the network can ignore many distractor objects rather than how small the target is; decoupling object size from clutter density would settle which variable drives the floor.","If the floor is largely a crowding effect, then a detection module that first proposes candidate locations could push the O2I limit lower without additional data, since the classifier would no longer need to aggregate over thousands of irrelevant patches.","The saliency results already show localization quality degrades faster than classification at small O2I, suggesting that the limit may be even lower for weakly supervised detection than for classification; that gap deserves its own scaling curve.","For pathology, the scaling result implies that weakly supervised whole-slide classification at clinically realistic O2I levels would need tens of thousands of slides, so the practical bottleneck is data-efficient inductive biases rather than raw dataset collection."],"forward_implications":["At any fixed training-set size there is a smallest object size for which an image-level CNN can classify better than chance, so practitioners working with very small targets should expect random performance unless they can grow the dataset.","Dataset collection effort must grow much faster than the reduction in object size: a small further decrease in the object-to-image ratio demands a large increase in training examples to hold accuracy.","Raising model capacity improves low-signal generalization, but the benefit shrinks when the training set is small, so capacity alone is not a substitute for data.","If an approximate object size is known, setting the receptive field to cover it is a useful inductive bias; the wrong pooling choice can hurt optimization mainly in the very low signal-to-noise regime."],"supporting_citations":[{"why":"Supplies the 28-by-28 handwritten digit images used to build the nMNIST dataset, including the digit 3 as the target and the remaining digits as clutter.","marker":"Lecun et al., 1998"},{"why":"Provides the gigapixel histopathology slides and pixel-level lesion annotations from which the nCAMELYON crops are extracted.","marker":"Ehteshami Bejnordi et al., 2017"},{"why":"Introduces the cluttered-MNIST recipe of placing digits on a larger canvas, the template nMNIST extends.","marker":"Ba et al., 2015"},{"why":"Contributes the convolutional backbone with explicit receptive-field control that all pipeline variants use.","marker":"Brendel & Bethge, 2019"},{"why":"Supplies the random-label memorization experiment that the optimization comparison uses to argue structured noise is harder than isotropic noise.","marker":"Zhang et al., 2017"},{"why":"Underpins the memorization framing that motivates the optimization difficulty analysis.","marker":"Arpit et al., 2017"},{"why":"Provides the soft-attention pooling formulation tested as one of the four global pooling operations.","marker":"Ilse et al., 2018"}],"fun_headline_variants":["Tiny objects demand a steep data tax on CNNs","CNN generalization on small targets requires soaring data","Smaller objects inflate training data needs exponentially","Object-to-image ratio sets a hard floor for CNN learning","For tiny objects, data hunger grows inversely with size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the failure threshold is controlled by the object's share of the image, but the main testbed changes the number of distractor digits together with the canvas size, so the claimed scaling could be driven by crowding rather than by object size.","fun_headline_variants_meta":{"raw":{"variants":["Tiny objects demand a steep data tax on CNNs","CNN generalization on small targets requires soaring data","Smaller objects inflate training data needs exponentially","Object-to-image ratio sets a hard floor for CNN learning","For tiny objects, data hunger grows inversely with size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3275,"prompt_tokens":902,"completion_tokens":2373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2298}},"tokens_in":518,"tokens_out":2373,"duration_ms":17563,"temperature":1.0,"reasoning_tokens":2298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:57:39.521690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1024-by-1024 digit experiment with the same 0.075% object-to-image ratio but with, say, 10 clutter digits instead of 400; if the network then classifies above chance, the O2I limit is really a clutter-count limit, and the paper's headline scaling would not transfer to constant-clutter settings.","supporting_citations":[{"cited_title":"Approximating CNN s with bag-of-local-features models works surprisingly well on imagenet","cited_arxiv_id":null,"evidence_quote":"Contributes the convolutional backbone with explicit receptive-field control that all pipeline variants use."}],"review_version":1}