{"id":"50faab1c-462c-45d9-8608-8f045294d5c7","arxiv_id":"1908.00704","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A greedy breadth-first search over augmentation operations, with probabilities applied later via Pareto weighting, finds effective policies with far less search compute than AutoAugment.","lead":"Greedy AutoAugment replaces AutoAugment's expensive reinforcement-learning search with a greedy breadth-first search over image transformations. The authors report better classification accuracy on most of four datasets and eleven networks, while using about 360 times less search compute, a practical result for limited compute budgets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Five-epoch child-network training is the load-bearing assumption: the paper never shows that 5-epoch validation rankings predict 200-epoch final rankings, so the greedy prefix selection may be selecting noise.","rationale":"The reader's weakest-assumption analysis identifies exactly the point I consider most load-bearing: the entire greedy selection procedure is built on 5-epoch child-network scores, and the paper provides no correlation or sensitivity evidence that these scores are reliable proxies for final performance. I agree with this assessment, and I do not think my review moves the verdict: the paper should remain CONDITIONAL, with acceptance contingent on demonstrating that the 5-epoch proxy is predictive or on replacing it with a validated proxy. I considered other concerns such as the lack of error bars, the use of transferred AutoAugment policies rather than a same-cost search, and the aggregate accuracy claims, but those are secondary to the internal validity of the search mechanism. Even if the baselines were perfectly fair, the method's claimed contribution would fail if the greedy expansions are made on noisy rankings. The proposed test directly checks the predictive relationship between short and long training, which is the omitted piece of evidence in Section 5.2.","tokens_in":12514,"tokens_out":5912,"duration_ms":62783,"concrete_test":"On one dataset/network pair (e.g., CIFAR-10 with ResNet18), fix the candidate set of 200 single-operation policies from Algorithm 1's first layer and train each candidate for 5, 20, 50, and 200 epochs using identical hyperparameters and data splits. Compute the Spearman rank correlation between the 5-epoch validation scores and the 200-epoch final validation scores. Then run Algorithm 1 using 5-epoch and 50-epoch child training and compare the final 200-epoch test accuracies of the selected policies. If the 5-epoch ranking does not correlate with the 200-epoch ranking, or if the 5-epoch search selects policies significantly worse than the 50-epoch search, the greedy mechanism is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The greedy selection mechanism in Algorithm 1 is entirely driven by validation scores from child networks trained for only 5 epochs, as stated in Section 5.2: 'The number of epochs used for our child networks was only 5 epochs.' The paper offers no evidence that the ranking of candidate policies at 5 epochs is predictive of rankings after the final 200-epoch training, and no variance or seed-dependence analysis is reported for these scores. Because lines 1-23 keep only the single best-scoring prefix at each depth, a noisy early evaluation permanently discards all unselected branches. If 5-epoch scores are dominated by optimization noise rather than by policy quality, the selected sub-policies are effectively arbitrary prefixes, and any reported accuracy gains would have to be attributed to the later Pareto weighting (Eq. 2) or to dataset-specific search itself, not to the greedy breadth-first search. This is the weakest link in the causal chain from the proposed search to the headline claims of better accuracy and a 360x compute reduction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Greedy AutoAugment, a data-augmentation policy search algorithm that replaces the RNN controller of AutoAugment with a greedy breadth-first search over augmentation operations. During the search the application probability is fixed to one, and after the search a Pareto-weighted random selection scheme (Eq. 2) sets the probabilities for the d selected policies. The authors report experiments on Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN with eleven network architectures, claiming that Greedy AutoAugment improves classification accuracy while using 360 times fewer computational resources than AutoAugment. The central claims are the accuracy improvement and the compute-reduction factor of 360.","tokens_in":12727,"tokens_out":9001,"duration_ms":75799,"significance":"If the central claims are correct, the paper provides a useful and conceptually simple baseline for data-augmentation policy search: fixing probabilities during search and deferring them to a post-hoc weighting step is a sensible way to reduce the search cost, and the use of a greedy expansion is a reasonable alternative to reinforcement-learning controllers. The paper tests on a broad range of architectures and four datasets, and it builds on public implementations of the augmentation operations. However, the strength of the evidence is currently insufficient. The AutoAugment baseline is not re-searched under matched conditions, no uncertainty quantification or significance testing is reported, the five-epoch child-network scores that drive the greedy search are not validated against final 200-epoch accuracies, and the compute-reduction claim rests on unverified assumptions about child-network cost. The conceptual contribution is real, but the experimental validation needs substantial additional work before the headline claims can be accepted.","major_comments":[{"comment":"The greedy selection in Algorithm 1 is driven entirely by validation scores from child networks trained for only five epochs, as stated in Section 5.2, but the final evaluation is after 200 epochs. The manuscript provides no evidence that the ranking of candidate policies at five epochs predicts the ranking after full training, and no variance or seed-dependence analysis of the five-epoch scores is reported. Because lines 9 and 22 keep only the single best-scoring prefix at each depth, any noise in the five-epoch evaluation permanently discards all unselected branches; if these scores are dominated by optimization noise, the selected sub-policies are effectively arbitrary and the later Pareto weighting in Eq. (2), rather than the greedy search, would be the source of any accuracy gains. This is the load-bearing assumption behind the central accuracy claim and needs to be tested, for example by correlating five-epoch and 200-epoch rankings on a subset of candidates.","section":"Section 5.2, Algorithm 1"},{"comment":"The comparison against AutoAugment is not a controlled comparison of the two search algorithms. For CIFAR-10, CIFAR-100, and SVHN the authors apply the published AutoAugment policies, and for Tiny ImageNet they apply the ImageNet policy, so AutoAugment is never re-searched under the same conditions (same child networks, same epochs, same data splits). The claim in Section 5 that this is fair because AutoAugment policies are transferable is an assumption, and the Tiny ImageNet results in Table 2 actually show that the transferred ImageNet policy hurts accuracy, which undermines that assumption. The 360x compute reduction in Section 5.2 is computed as (15000*120)/(1000*5) under the explicit assumption that child networks are interchangeable, but no wall-clock or GPU-hour measurements are reported; moreover, if a separate Greedy AutoAugment search is run for each of the eleven architectures, the 1000-trial figure would need to be multiplied by eleven, changing the claimed factor.","section":"Section 5.1 and Section 5.2"},{"comment":"The accuracy results are reported as averages over five trials without standard deviations, confidence intervals, or significance tests. The headline aggregate figures such as '78.34% higher accuracy' and '25.60% higher accuracy' are sums of percentage-point differences across eleven networks, which is not a well-defined performance metric because it treats a large gain on one network and a small loss on another as equivalent. The authors should report per-network uncertainty and test whether the differences between Greedy AutoAugment and the two baselines are statistically significant.","section":"Table 2, Section 5.1"},{"comment":"The Pareto weighting scheme introduces free parameters alpha=2, d=25, and epsilon=5, and these values are stated without any sensitivity analysis. The final accuracy is the result of the entire pipeline, so the authors should show that the reported gains are stable with respect to reasonable variations of these parameters; otherwise the improvement over the manual baseline could be an artifact of a particular post-hoc selection rule rather than of the greedy search.","section":"Section 4, Eq. (2)"},{"comment":"The claimed reduction from exponential to linear search space is not clearly demonstrated. Eq. (1) is typeset as an unclear summation, and the GAutoAugment column of Table 3 lists 200, 4200, 8200, 12200 trials for l=1..4 without explaining the formula from which these numbers follow; the text in Section 5.2 says 'the number of possible trials is only 4000,' which does not match any single row. The 360x claim relies on this accounting, so the authors need to state precisely how the trial counts are derived and whether they include the cost of separate searches for each dataset and each architecture.","section":"Section 3, Eq. (1), Table 3"}],"minor_comments":[{"comment":"There are multiple typos, for example 'artifitial' in the abstract and 'the the arxiv version' in the footnote on page 1.","section":"Abstract and Introduction"},{"comment":"The GAutoAugment columns are labeled 'Cifar10 Cifar10 SVHN' instead of 'Cifar10 Cifar100 SVHN'; the corresponding values appear to be CIFAR-100 results, so the header should be corrected.","section":"Table 2"},{"comment":"Table 1 contains several misspellings: 'Filliping' for both FlipLR and FlipUD, 'Redicing' for Posterize, and 'Sheering' for ShearX and ShearY.","section":"Table 1"},{"comment":"The running head on pages 2-8 reads 'Auto-Augmentation for n-Shot Learning,' which does not match the paper's title, 'Greedy AutoAugment.'","section":"Running head"},{"comment":"Phrases such as 'AutoAugment reduced the accuracy with 52.73%' and 'the proposed method provided 78.34% higher accuracy' should be phrased as sums of percentage-point differences or as per-network results, since the current wording implies a relative accuracy change that is not what is computed.","section":"Section 5.1"},{"comment":"The relationship between the search split (Xtr, Xte) and the final training is under-specified: the text does not state whether the final training uses the full original training set or only the Xtr portion after the Xte split used for scoring.","section":"Algorithm 1 and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv version of a paper that, according to the reference list, has appeared in Pattern Recognition Letters. The editorial office may wish to verify the submission status and whether this version is intended for review as a new submission. The experimental protocol, especially the lack of a re-searched AutoAugment baseline and the absence of uncertainty quantification, would need to be substantially strengthened before the claims are publishable in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Greedy AutoAugment is a sensible engineering simplification of AutoAugment: replace the RL controller with a greedy breadth-first search, fix the probability to 1 during search, then weight the selected policies with a Pareto distribution. The core idea is easy to understand and implement, and the paper is clearly written. The experimental claims are plausible but overstated; the useful core should survive a reworked evaluation.\n\nWhat is genuinely new is the specific combination of fixed probability during search and Pareto weighting after search. The comparative results on 11 architectures and 4 datasets show many networks improving, and it is a fair observation that transferred AutoAugment policies can hurt on a subset like Tiny ImageNet. This is a practical paper, not a theoretical one.\n\nThe soft spots are in the evaluation. The AutoAugment baseline is not re-searched under the same conditions; it uses transferred policies from the original paper, so the comparison is not like-for-like. The aggregate accuracy claims are sums of per-network percentage-point differences, which inflate the headline. No error bars or significance tests are reported, even though the results are said to be averages of five trials. The compute comparison assumes child networks are interchangeable between both methods, which is generous. The biggest concern is the 5-epoch child network training: the entire greedy selection is based on validation accuracy after five epochs, and the paper gives no evidence that the ranking at five epochs predicts the ranking after the final 200-epoch training. That is a load-bearing assumption, and it is not tested. Some networks do worse with the method, and those failures are not discussed.\n\nThe paper would be useful for anyone who wants a cheap, simple augmentation search to use on modest hardware. It deserves a serious referee, but the authors need to redo the comparison with the same search conditions for both methods, report variance, and add a sanity check for the 5-epoch proxy. Without that, the 360x claim should not be taken at face value.","headline":"A sensible greedy simplification of AutoAugment with plausible but overstated accuracy and compute claims; the 5-epoch child training proxy is the key untested assumption.","tokens_in":13241,"tokens_out":3985,"would_cite":false,"duration_ms":36239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A greedy, linear-cost search over image augmentations finds better policies than AutoAugment at 1/360th the compute.","keywords":["data augmentation","greedy search","AutoAugment","image classification","neural networks","policy search","Pareto distribution"],"falsifier":"Run the greedy search but train each child network for the full 200 epochs and compare the selected policies with those from 5-epoch scoring; if the rankings do not match on a held-out set, the reported gains come from the Pareto weighting rather than from the greedy search itself.","tokens_in":12296,"feed_emoji":"🖼️","tokens_out":4703,"duration_ms":42455,"temperature":0.7,"pith_summary":"The paper claims that the expensive reinforcement-learning search behind AutoAugment can be replaced by a greedy breadth-first search that grows the augmentation policy one operation at a time, with the probability fixed to one during search. This turns an exponential search space into a linear one and makes per-dataset augmentation search affordable. On Tiny ImageNet, CIFAR-10, CIFAR-100, and SVHN, the authors report that the resulting policies beat both manual augmentation and transferred AutoAugment policies on most of eleven architectures while using 360 times fewer computational resources.","feed_headline":"Greedy search finds augmentations 360x cheaper","feed_subtitle":"A linear-growth search beats transferred AutoAugment policies on four image benchmarks.","key_machinery":"The engine is Algorithm 1, a greedy breadth-first search over (technique, magnitude) pairs with probability set to 1. At each layer, the best previous policy is extended by every operation and magnitude, and each candidate is scored by the validation accuracy of a child network trained for only 5 epochs. After $k$ layers, the $d$ best policies are selected, and their application probabilities are assigned post hoc using a Pareto distribution $v_i = (1/i)^\\alpha$, giving higher weight to better-scoring policies and to the original data. This reduces the search space from $(t_n \\times p_n \\times m_n)^l$ to $k \\times (t_n \\times m_n)$.","core_discovery":"The central claim is that a simple greedy search over augmentation operations and magnitudes, with probability ignored during search and applied afterward via a Pareto-weighted distribution, finds better augmentation policies than policies transferred from AutoAugment's separate search, at a fraction of the cost. Specifically, the search evaluates only 1000 child networks trained for 5 epochs each, versus AutoAugment's 15000 child networks trained for 120 epochs, a 360-fold reduction. Against transferred AutoAugment policies, Greedy AutoAugment improves accuracy on 9 of 11 networks on Tiny ImageNet, 9 of 11 on CIFAR-10, 9 of 11 on CIFAR-100, and 10 of 11 on SVHN, while manual augmentation also outperforms transferred AutoAugment on several datasets.","pith_inferences":["If five-epoch rankings are stable across training lengths, the same greedy scheme could be extended to longer policy chains or larger operation sets at modest cost.","The Pareto weighting parameter $\\alpha = 2$ is chosen without an ablation study; future work could test whether other values trade off exploration of weaker policies differently.","Because the search relies on validation accuracy after 5 epochs, part of the reported gain could come from the Pareto weighting rather than the greedy selection; an ablation with random policy prefixes would isolate the search's contribution.","The operation set and scoring procedure could carry over to other vision tasks such as object detection or segmentation, where augmentation policies are still hand-designed."],"forward_implications":["Augmentation policy search becomes cheap enough to run per dataset, removing the need to transfer policies across datasets.","The search cost grows linearly in the number of augmentation layers instead of exponentially, so deeper policies become accessible.","Transferred AutoAugment policies are not universally beneficial: on Tiny ImageNet they hurt all tested networks, while per-dataset greedy search helps most.","The Pareto weighting lets a single search pass output multiple candidate policies with a principled probability assignment."],"supporting_citations":[{"why":"Defines AutoAugment, the baseline method whose search space and transferred policies Greedy AutoAugment is compared against.","marker":"Cubuk et al., 2018"},{"why":"Supplies the NASNet controller and policy-gradient mechanism that Greedy AutoAugment replaces with greedy breadth-first search.","marker":"Zoph et al., 2018"},{"why":"Provides the Pareto distribution used to assign application probabilities to the selected policies after search.","marker":"Arnold, 2015"},{"why":"Supplies the network implementations used for uniform evaluation across all compared methods.","marker":"Liu"},{"why":"Introduces the Tiny ImageNet dataset used in the accuracy comparisons.","marker":"Le and Yang, 2015"},{"why":"Introduces the CIFAR-10 and CIFAR-100 datasets used in the experiments.","marker":"Krizhevsky et al., 2009"},{"why":"Introduces the SVHN dataset used in the experiments.","marker":"Netzer et al., 2011"}],"fun_headline_variants":["Greedy search cuts AutoAugment cost 360x","Linear search outperforms AutoAugment on four benchmarks","Greedy AutoAugment: 360x less compute, higher accuracy","Greedy beats transferred policies on 9 of 11 networks","No probability needed: greedy wins on four datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selection of policies rests on the assumption that the validation accuracy of a child network after only 5 epochs ranks augmentation policies in the same order as the full 200-epoch training.","fun_headline_variants_meta":{"raw":{"variants":["Greedy search cuts AutoAugment cost 360x","Linear search outperforms AutoAugment on four benchmarks","Greedy AutoAugment: 360x less compute, higher accuracy","Greedy beats transferred policies on 9 of 11 networks","No probability needed: greedy wins on four datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1706,"prompt_tokens":843,"completion_tokens":863,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":778}},"tokens_in":459,"tokens_out":863,"duration_ms":7461,"temperature":1.0,"reasoning_tokens":778,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:37:03.907967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the greedy search but train each child network for the full 200 epochs and compare the selected policies with those from 5-epoch scoring; if the rankings do not match on a held-out set, the reported gains come from the Pareto weighting rather than from the greedy search itself.","supporting_citations":[],"review_version":1}