{"id":"acd02741-6862-4ca4-bc42-3419474b28be","arxiv_id":"1908.03835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AutoGAN applies reinforcement-learning-based neural architecture search to GAN generators, discovering a CIFAR-10 architecture with FID 12.42 and an STL-10 FID 31.01, both state of the art in 2019.","lead":"This paper introduces AutoGAN, the first neural architecture search method for generative adversarial networks, and shows that automatically discovered generator architectures beat many human-designed ones on standard image benchmarks. It matters because it demonstrates that the expert work of designing GAN backbones can be automated, achieving state-of-the-art FID scores on CIFAR-10 and STL-10 at publication time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unsearched discriminator growth routine is underspecified and load-bearing; if the reported FID shifts under alternative D schedules, the SOTA claim cannot be attributed to the NAS search alone.","rationale":"The paper is a good-faith first NAS-for-GANs study with public code, component ablations, a random-search comparison, and a transfer experiment to STL-10 that provides independent support. However, the headline FID SOTA numbers depend on a discriminator growth routine that is referenced but not specified in the manuscript (Section 3: 'The details of growing D will be explained more in the supplementary'; no supplementary is present in the arXiv v1). Because the paper explicitly argues that G/D capacity balance is essential and rejects other G/D search configurations, the fixed D-growth rule is not a peripheral implementation detail: it is part of what makes the search work. Without a sensitivity check or a full specification, the central claim cannot be fully separated from this unsearched component. This matches the reader's weakest assumption, so I agree with the reader's assessment. The concern is addressable and does not by itself invalidate the paper, so the CONDITIONAL verdict should stand; I recommend no change to the reader's verdict.","tokens_in":13316,"tokens_out":6525,"duration_ms":75928,"concrete_test":"Run the released AutoGAN code on CIFAR-10 under at least two alternative discriminator growth schedules, for example: (a) keep D at the final depth from the start, and (b) grow D channels twice as fast as the default, while keeping the generator search space, controller, reward, and compute budget fixed. For each schedule, record the top discovered generator architecture and retrain it from scratch with 5 seeds, reporting mean and standard deviation of FID. If the best FID differs by more than about 2 points across schedules, or the selected generator architecture changes materially, the unsearched D routine is load-bearing; if FID and architecture are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AutoGAN's search discovers generator architectures that beat hand-designed GANs on FID. The search optimizes only G, while D is grown by 'a given routine to stack pre-defined blocks' (Section 3), with details deferred to a supplementary that is absent from the arXiv v1. The paper itself identifies G/D capacity balance as critical: it rejects both fixed-D search and joint G/D search because of imbalance and instability. That makes the D-growth schedule load-bearing: it is the mechanism intended to keep training stable while the controller explores generator architectures. If the schedule is ad hoc, too strong, or too weak, the reward signal and the final FID could reflect the discriminator routine rather than the NAS method. The paper offers no ablation varying D growth and no specification of the routine in the main text. Section 5 concedes that discriminators are not searched, but that does not state what the fixed growth rule is. The SOTA claim should therefore be read as conditional on the exact, unreported D schedule until the released code or supplementary pins it down and a sensitivity check is run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AutoGAN introduces the first neural architecture search framework for generative adversarial networks. The authors define a generator cell search space over block type, normalization, upsampling, skip connections, and in-cell shortcuts, and train an RNN controller by REINFORCE using Inception Score as the reward. The search is accelerated by ENAS-style parameter sharing, a dynamic-resetting strategy for collapsed shared models, and a multi-level progressive architecture search with beam search. On CIFAR-10, the discovered generator reaches IS 8.55 and FID 12.42, and the same architecture transferred to STL-10 reaches IS 9.16 and FID 31.01. The paper also provides ablations comparing IS versus FID reward, dynamic resetting, MLAS versus SLAS, and random search baselines, and closes with a discussion of search-space, resolution, and discriminator-search limitations.","tokens_in":13532,"tokens_out":5969,"duration_ms":64580,"significance":"If the results hold, the paper is significant as a first proof-of-concept for architecture search in GANs and provides evidence that generator macro-architecture affects FID. The strengths include a detailed method description, a quantitative proxy-reward correlation (Spearman 0.779), an explicit comparison between IS-based and FID-based rewards, a speedup analysis for dynamic resetting, a comparison to random search, and a public code link. The reported FID improvements over hand-crafted baselines are potentially important. However, the central state-of-the-art claim currently rests on an underspecified discriminator growth routine and on a selection protocol that is not fully controlled; these issues should be addressed before the claim can be taken at face value.","major_comments":[{"comment":"The discriminator growth routine is load-bearing and underspecified. The paper states that D grows \"by following a given routine to stack pre-defined blocks\" and defers the details to a supplementary file; the arXiv v1 contains no supplementary, and no pinned commit of the released code is identified. Because the paper explicitly rejects both fixed-D search and joint G/D search on capacity-imbalance grounds, this D schedule is the mechanism that is supposed to keep the shared GAN trainable during the search. Without a precise specification of the routine, or an ablation or sensitivity study over different D growth choices, the reported FID gains cannot be attributed to the NAS search of G as opposed to the hand-designed discriminator schedule. Please provide the full routine, a citation if it is a standard one, and a sensitivity check.","section":"Section 3, 'Technical Approach' (first paragraph)"},{"comment":"The final architecture is selected after retraining the top-K candidates by highest Inception Score, so the reported IS of 8.55 is a selected maximum and the IS comparison in Table 1 is partly by construction. The corresponding FID values of the top-1, top-2, and top-3 candidates differ (12.42, 13.67, 13.87), but no repeated-search or retraining-seed intervals are provided for any FID number. To support the state-of-the-art FID claim, the paper should report the distribution of FID over search repetitions or at least over final retraining seeds, and should report FID for all derived candidates rather than only for the IS-selected architecture.","section":"Section 3.3.3, 'Architecture Derivation'; Table 1"},{"comment":"The comparison baselines are collected from their original papers and may therefore use different FID evaluation protocols. The manuscript does not state the number of generated samples used for FID computation, the reference implementation/version, or any confidence intervals for FID. Since the central claim is a new state-of-the-art FID, the evaluation protocol should be stated precisely and, ideally, the baselines should be recomputed under the same protocol. At minimum, the FID variance over seeds must be reported.","section":"Section 4.1, Table 1; Section 4.2, Table 2"},{"comment":"The STL-10 result uses the CIFAR-10-discovered architecture without re-searching on STL-10; it is an impressive transfer result but not an STL-10 search result. The abstract's phrasing \"setting new state-of-the-art FID scores ... 31.01 on STL-10\" should be qualified as \"with a transferred architecture,\" and the comparison to methods whose architectures may have been tuned on STL-10 should be framed as a transferability study rather than a direct search comparison.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"There are several typos: \"avaliable\" in the abstract, \"CFIAR-10\" in Section 4.3.1, \"unlabled\" in Section 4.2, and \"performace\" in the Figure 9 caption.","section":"Abstract and Section 4.3.1"},{"comment":"The claim that IS-driven and FID-driven searches show \"comparable performance\" is supported only by plots; please report the quantitative final IS and FID values for both searches so the reader can judge the size of the difference.","section":"Section 4.3.2, Figure 8"},{"comment":"The MLAS versus SLAS ablation reports only Inception Score; since the paper's headline metric is FID, please report FID for both search strategies as well.","section":"Section 4.3.4, Figure 10"},{"comment":"The statement that the search space \"can only cover SN-GAN\" is not fully explained; since spectral normalization is applied only to the discriminator in this paper, please clarify why SN-GAN's generator is covered by the search space while the other compared methods are not.","section":"Section 1 and Section 4.1"},{"comment":"The pseudo-code's signature \"train(generator,discriminator,FDR)\" is slightly ambiguous because FDR is both an input and an output of the training procedure; please make the data flow of the dynamic-resetting flag explicit.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is reproducibility: the missing supplementary and unpinned code make the discriminator growth routine, which is central to the method's stability, unverifiable in the current version. The novelty claim as the first NAS for GANs appears appropriate, and the paper fits the venue's scope, but the empirical claims need the additional controls described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is the first real attempt to run NAS inside GAN training, and it mostly works. The authors define a search space over the generator, use an RNN controller with parameter sharing, add a dynamic-resetting trick to avoid mode collapse during search, and search in stages. The discovered generator on CIFAR-10 gets a 12.42 FID, which was SOTA at the time, and it transfers to STL-10 with a 31.01 FID. Those numbers are credible, not obviously cherry-picked: the paper includes a proxy-reward correlation (Spearman 0.779), a comparison between IS-based and FID-based rewards, an ablation of dynamic resetting, a comparison of multilevel vs single-level search, and random search baselines. The code is public.\n\nWhat I find genuinely good beyond the results: the paper is honest about its own limits. The conclusion explicitly says the search space is too narrow, that discriminators are not searched, and that higher-resolution synthesis is untested. The architecture it discovers also lines up with what people already suspected—pre-activation blocks, nearest upsampling over deconv, no norm, dense skip paths—so the search isn't discovering nonsense.\n\nNow the soft spots, in proportion. The most load-bearing is the discriminator growth routine. The search only optimizes G; D is grown by 'a given routine' whose details are deferred to a supplementary that is not on arXiv v1. The authors themselves say G/D capacity balance is critical, and they reject two-way search because it's unstable. That makes the fixed D schedule part of the recipe, and if the results shift under a different schedule, the SOTA attribution to the NAS method weakens. That's a real gap, but it's not fatal: the random search baselines and the ablations all use the same D schedule, so the relative comparisons hold at least as a controlled experiment. What is missing is a sensitivity check varying D growth, and we don't know the exact routine from the main text. The referee should demand that.\n\nAlso minor: the headline FID numbers have no error bars, and the final architecture is selected on Inception Score, so the reported IS of 8.55 is partly fitted. But the FID-reward search gives comparable results, which mitigates the selection-bias worry. I'd like to see the top-3 spread, which they do report (FID 12.42, 13.67, 13.87), so the top1 isn't an outlier.\n\nOverall: this is a solid first step in a new subfield. It deserves a serious referee, and the revision should pin down the D-growth routine, add error bars, and ideally run a small sensitivity study on D growth. I would bring it to reading group and would cite it as the first NAS-for-GANs work.","headline":"First credible NAS-for-GANs paper; results are solid but the SOTA claim rests on an underspecified discriminator-growth routine that peer review should pin down.","tokens_in":14086,"tokens_out":2636,"would_cite":true,"duration_ms":26042,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoGAN extends neural architecture search to GANs, searching only the generator and reporting state-of-the-art FID scores of 12.42 on CIFAR-10 and 31.01 on STL-10.","keywords":["AutoGAN","neural architecture search","generative adversarial networks","generator architecture","Inception score reward","Fréchet inception distance","multi-level architecture search","parameter sharing"],"falsifier":"Retrain the discovered CIFAR-10 generator and the strongest hand-crafted baselines from scratch under matched training budgets, seeds, and hyperparameters; if the FID gap to 12.42 does not reproduce or falls within run-to-run noise, the search advantage is not established. Alternatively, rerun the AutoGAN search with a different fixed discriminator growth routine; if FID worsens substantially, the discriminator schedule is doing the work.","tokens_in":47,"feed_emoji":"🧠","tokens_out":6539,"duration_ms":124808,"temperature":0.7,"pith_summary":"The paper tries to establish that the architecture of a GAN generator can be discovered by neural architecture search rather than designed by hand, and that the discovered generator beats hand-crafted ones. It introduces AutoGAN, in which an RNN controller selects each cell's convolution block, normalization, upsampling type, and skip connections while the discriminator follows a fixed growth routine. Using Inception score as the reward, parameter sharing, dynamic resetting, and progressive cell-by-cell search, AutoGAN reports a CIFAR-10 FID of 12.42 and an STL-10 FID of 31.01, both state-of-the-art at publication, with competitive Inception scores. If correct, this opens GAN generator design to automation the way classification architectures were automated earlier.","feed_headline":"Search finds GAN generator with record FID 12.42","feed_subtitle":"First NAS for GANs; discovered generator tops rivals on FID, 12.42 on CIFAR-10 and 31.01 on STL-10.","key_machinery":"The load-bearing object is the per-cell search-space tuple $(skip_1,\\dots,skip_s,C,N,U,SC)$, where $skip_i$ enables a connection from cell $i-1$, $C$ is a pre- or post-activation convolution block, $N$ is batch norm, instance norm, or none, $U$ is bilinear, nearest-neighbor, or deconvolution upsampling, and $SC$ is an in-cell shortcut. An LSTM controller emits these choices autoregressively; its hidden state is reused in the next search stage, and top-$K$ beams continue to the next cell. Parameter sharing lets many sampled child generators share weights during search, dynamic resetting reinitializes shared GAN parameters when loss standard deviation falls below a threshold, and multi-level search grows cells bottom-up. The reward for each sampled architecture is the Inception score, computed cheaply on the shared model.","core_discovery":"AutoGAN's central discovery is that a search algorithm can rival, and on FID surpass, hand-crafted generator architectures in unconditional image generation. The controller builds generators cell by cell from a search space of block type, normalization, upsampling, in-cell shortcut, and skip connections to earlier cells; it is trained by REINFORCE with Inception score as reward. A parameter dynamic-resetting strategy restarts the shared GAN when training-loss variance collapses, and a multi-level strategy searches cells progressively with beam search. On CIFAR-10 the found generator reaches Inception score 8.55 and FID 12.42, improving on prior FID results. The same architecture retrained on STL-10 gives FID 31.01, also state-of-the-art, and the paper's ablations show its proxy correlates with true Inception score (rank correlation 0.779), that FID as reward gives comparable results, and that multi-level search and dynamic resetting each help.","pith_inferences":["An implicit test the paper does not run: replace the fixed discriminator growth routine with a different one and re-run the search; equal FID would strengthen the generator-search claim, while a drop would suggest the discriminator schedule does much of the work.","Because the search space includes SN-GAN but excludes attention, style-based, and multi-discriminator generators, the comparison is to a subset of hand-crafted GANs; searching with those blocks as candidates would test whether the search advantage persists outside the covered space.","The reported 43-hour CIFAR-10 search cost suggests a direct extension: search at lower resolution or with more aggressive weight sharing, then fine-tune or transfer the discovered generator to higher resolution, mirroring what classification NAS has done.","The discovered preferences—pre-activation blocks, nearest or bilinear upsampling over deconvolution, no normalization, and dense skip connections—could be used as a prior to shrink the search space in future work, but the paper does not claim this."],"forward_implications":["A discovered generator architecture, not a new loss or training trick, is enough to set new state-of-the-art FID on CIFAR-10 and STL-10.","GAN generator search is feasible without searching the discriminator: fixing D to a growing routine with spectral normalization keeps training stable enough for reinforcement-learning-based NAS.","The CIFAR-10-found architecture transfers to STL-10, suggesting searched GAN generators can generalize across datasets and resolutions.","An Inception-score proxy with parameter sharing is a workable reward: proxy and true Inception scores correlate with rank correlation 0.779, and searching with FID as reward reaches comparable quality.","Multi-level progressive search outperforms single-level search both in final Inception score and in training time."],"supporting_citations":[{"why":"Supplies the RNN-controller framework for generating architectures that AutoGAN adapts to GANs.","marker":"[71]"},{"why":"Provides the parameter-sharing strategy that AutoGAN extends with dynamic resetting.","marker":"[46]"},{"why":"Defines Inception score, the reward used to update the controller.","marker":"[50]"},{"why":"Defines FID, the evaluation metric in which AutoGAN reports state-of-the-art results.","marker":"[18]"},{"why":"Motivates progressive growing and serves as a strong hand-crafted baseline.","marker":"[26]"},{"why":"Provides the spectral-normalization training setting and a hand-crafted baseline covered by the search space.","marker":"[42]"},{"why":"Supplies the multi-level and beam-search strategy used in progressive cell search.","marker":"[37]"},{"why":"Gives the REINFORCE update used to train the controller.","marker":"[59]"},{"why":"Provides the random-search baselines that the ablation compares against.","marker":"[34]"}],"fun_headline_variants":["AutoGAN: Neural search discovers GAN that beats hand-crafted","NAS-designed GAN tops FID on CIFAR-10 with 12.42","AutoGAN: Search finds GAN generator with state-of-the-art FID","First NAS for GANs yields generator that outscores rivals","AutoGAN: AI search designs GAN, sets new FID benchmark"],"cache_read_input_tokens":16256,"weakest_assumption_plain":"The central assumption is that the fixed discriminator-growth routine keeps the generator/discriminator capacity balance intact; if that routine is unbalanced or ad hoc, the reported gains may be due to the discriminator schedule rather than to the searched generator, and the paper leaves the routine's details to the supplementary.","fun_headline_variants_meta":{"raw":{"variants":["AutoGAN: Neural search discovers GAN that beats hand-crafted","NAS-designed GAN tops FID on CIFAR-10 with 12.42","AutoGAN: Search finds GAN generator with state-of-the-art FID","First NAS for GANs yields generator that outscores rivals","AutoGAN: AI search designs GAN, sets new FID benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2732,"prompt_tokens":961,"completion_tokens":1771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1673}},"tokens_in":577,"tokens_out":1771,"duration_ms":13579,"temperature":1.0,"reasoning_tokens":1673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:00:25.874223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the discovered CIFAR-10 generator and the strongest hand-crafted baselines from scratch under matched training budgets, seeds, and hyperparameters; if the FID gap to 12.42 does not reproduce or falls within run-to-run noise, the search advantage is not established. Alternatively, rerun the AutoGAN search with a different fixed discriminator growth routine; if FID worsens substantially, the discriminator schedule is doing the work.","supporting_citations":[{"cited_title":"Efficient Neural Architecture Search via Parameter Sharing","cited_arxiv_id":null,"evidence_quote":"Provides the parameter-sharing strategy that AutoGAN extends with dynamic resetting."},{"cited_title":"Improved techniques for training gans","cited_arxiv_id":null,"evidence_quote":"Defines Inception score, the reward used to update the controller."},{"cited_title":"Progressive growing of gans for improved quality, stability, and variation","cited_arxiv_id":null,"evidence_quote":"Motivates progressive growing and serves as a strong hand-crafted baseline."},{"cited_title":"Progressive Neural Architecture Search","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-level and beam-search strategy used in progressive cell search."},{"cited_title":"Simple statistical gradient-following al- gorithms for connectionist reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Gives the REINFORCE update used to train the controller."},{"cited_title":"Random search and repro- ducibility for neural architecture search","cited_arxiv_id":null,"evidence_quote":"Provides the random-search baselines that the ablation compares against."}],"review_version":1}