{"id":"73cd0a97-3c49-4f77-8d2d-74524f39487a","arxiv_id":"1908.10714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"The paper reports that an automated forward thinking algorithm achieves about 97.5 percent test accuracy on MNIST, higher than the other architecture design methods it compares.","lead":"This bachelor thesis benchmarks search-based and dynamic neural architecture design methods on MNIST, and introduces an automated forward thinking algorithm that builds networks layer by layer. It is a readable empirical comparison for anyone interested in reducing manual effort in deep learning model design.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that automated forward thinking outperforms every investigated MLP rests on an invalid comparison: baselines were selected using the test set (Sec. 4.2.2 and 4.2.3) and the best AFT result is a best-of-20 test-set choice (Sec. 4.3.3).","rationale":"The reader's weakest assumption identifies test-set leakage in the baseline search algorithms as the main threat to the central claim. My reading agrees: Sec. 4.2.2 and Sec. 4.2.3 explicitly use the test set during random and evolutionary search, contradicting the stated protocol in Sec. 4.1.2. This makes the baseline numbers optimistic and the comparison unfair to the validation-driven AFT construction. I add one detail the reader did not emphasize: the AFT headline result of 97.86% is itself selected from 20 runs on the test set, so the 'best architecture' is also an optimistic estimator. The average AFT accuracy, 97.54% with standard deviation 0.17%, is close to the reported baseline means, so without a clean protocol the claimed advantage could be a selection artifact. The paper is honest about its preliminary scope, and the proposed check is a straightforward re-run that would settle the comparison. Since the reader already assigned CONDITIONAL on essentially this concern, my stress-test does not move the verdict.","tokens_in":34421,"tokens_out":4035,"duration_ms":43995,"concrete_test":"Re-run the full comparison under a single clean protocol: hold out the MNIST test set completely. For random search and evolutionary search, use validation-set accuracy for selection and fitness, train the selected architectures with early stopping on validation loss, and evaluate on the test set exactly once per selected architecture. For automated forward thinking, run the same 20 runs but choose the final architecture by validation accuracy (e.g., smallest architecture whose validation accuracy is within one standard error of the best validation architecture), then report the average and best test accuracy. If AFT's average or best test accuracy no longer exceeds the cleanly selected manual, random, and evolutionary baselines by a statistically significant margin, the paper's central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: automated forward thinking reportedly builds deeper networks than the search-based MLPs and achieves better test accuracy on MNIST. The comparison is only meaningful if every method's architecture is chosen without consulting the test set. The paper's own protocol states this in Sec. 4.1.2: 'The testing set will not be used until the final model architecture is decided on.' Yet Sec. 4.2.2 says random search networks 'will be assessed on the training set and the testing set' after five epochs, and Sec. 4.2.3 defines evolutionary fitness as 'the accuracy of the network on the testing set after training for three epochs.' These baselines are therefore optimistically biased: the reported best random/evolutionary MLPs were selected partly for test accuracy, so their advertised 97.2-97.3% figures are not clean generalization estimates. AFT, by contrast, selects candidate layers on validation accuracy. Comparing a test-selected baseline to a validation-selected construction inflates the apparent advantage of AFT. Additionally, AFT's headline 'best performing network architecture is [900, 600, 600, 300] with a testing accuracy of 97.86%' is a post hoc choice from 20 runs; the test set was used to pick this single architecture, and the average AFT accuracy is 97.54% with standard deviation 0.17%, which overlaps with the baselines' reported means when their own selection bias is considered. Thus the load-bearing premise that test data was not used to guide architecture selection is violated for the baselines and for the strongest AFT claim, so the central comparison is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, a bachelor thesis posted to arXiv, studies automated architecture design for feedforward neural networks. It surveys neural architecture search and constructive dynamic learning, then presents empirical comparisons on MNIST among manual search, random search, evolutionary search, several cascade-correlation variants (Cascor, Caser, CaserRe), forward thinking, and the author's proposed Automated Forward Thinking (AFT). AFT greedily adds hidden layers selected from a candidate pool of eight randomly sized layers, training each candidate for two epochs and choosing the one with highest validation accuracy. The paper reports that AFT builds deeper networks than the MLP baselines and achieves an average test accuracy of 97.54% (std 0.17%), with the best architecture [900,600,600,300] reaching 97.86%, and concludes that AFT outperforms every MLP investigated. The central empirical claim is currently undermined by the documented use of the test set during random and evolutionary search and by best-of-20 selection for the AFT headline number.","tokens_in":34730,"tokens_out":7929,"duration_ms":75163,"significance":"If the comparison were clean, the contribution would be a useful, simple constructive NAS method: AFT avoids end-to-end backpropagation through deep stacks, produces interpretable layer-by-layer training curves, and is cheap enough to run 20 times on a modest GPU. The paper also gives a clear negative result for cascade-correlation on high-dimensional MNIST and an interesting fix, CaserRe, which reuses output weights to reduce performance dips. Its strengths are the explicit protocol statement in Section 4.1.2, which makes the later violation easy to identify, the candid acknowledgment of the preliminary scope, and the transparent reporting of standard deviations. However, as written, the headline comparison is not trustworthy: the baselines are test-selected, the AFT best is a post-hoc test-set choice, and there is no statistical test. The contribution may be salvageable with a corrected protocol and additional benchmark tasks, but the current evidence is not sufficient to support the central claim.","major_comments":[{"comment":"The central comparison is invalidated by test-set leakage in the two search baselines. Section 4.1.2 states that the testing set will not be used until the final model architecture is decided on and that all model decisions will be based on validation and training data, yet Section 4.2.2 says random-search networks \"will be assessed on the training set and the testing set\" after training, and Section 4.2.3 defines evolutionary fitness as \"the accuracy of the network on the testing set after training for three epochs.\" The reported best random and evolutionary architectures (Tables 2, 4, 6) were therefore selected partly on test accuracy, making their 97.2–97.3% test figures optimistically biased estimates of generalization. Automated forward thinking, by contrast, selects candidate layers on validation accuracy (Section 4.3.3). Comparing a test-selected baseline with a validation-selected construction inflates the apparent advantage of AFT. The random and evolutionary searches should be re-run with all selection decisions made on the validation set, the test set held out until one final architecture per method is evaluated, and the comparison repeated with a corrected protocol.","section":"§4.2.2, §4.2.3, §4.1.2"},{"comment":"The headline AFT result is a best-of-20 test-set choice, not a clean out-of-sample estimate. The text reports that the \"best performing network architecture is [900, 600, 600, 300] with a testing accuracy of 97.86%\" after running the algorithm 20 times; Table 9 lists the ten smallest architectures with their Test Acc values, and the runs appear to be scored on the test set. Selecting the best of 20 architectures by test accuracy is itself a form of test-set model selection, and it will overestimate the performance a user would obtain by running AFT once and deploying the result. The average AFT test accuracy (97.54% ± 0.17%) is a more meaningful statistic, but even that average is not compared statistically with the baselines, and it overlaps with the baseline means once the baselines' selection bias is accounted for. The authors should pre-register a single architecture choice, or select the architecture on validation and evaluate it once on test, and should provide a paired or corrected significance comparison.","section":"§4.3.3, Table 9"},{"comment":"The claim that AFT \"yields better performance on the test data than any MLP investigated in this thesis\" rests on a single dataset (MNIST), one run each of the random and evolutionary search procedures, and no significance tests. Section 4.1.2 explicitly restricts the benchmark to MNIST, and Section 4.1 acknowledges that the results \"will not be sufficient to confirm or reject any hypotheses\"; Section 4.4 similarly calls the evidence preliminary. A single-dataset comparison with one realization of two baseline families cannot support a general superiority claim, especially when the reported differences are on the order of 0.2–0.3 percentage points. The manuscript should either restrict the conclusion to \"on MNIST under a corrected protocol, AFT performed comparably or slightly better\" or add additional benchmark tasks and multiple repeats of every search algorithm with validation-based selection.","section":"§4.1.2, §4.4"},{"comment":"The \"fully automated\" characterization is overstated. The AFT algorithm requires the user to set the candidate-pool size (8), the number of training epochs per candidate (2), the layer-width interval [50, 1000] in multiples of 50, the early-stopping patience, and, most restrictively, the constraint that layer widths must be subsequently decreasing, a choice the author says will be \"ommit[ted]\" from discussion. Section 4.3.4 acknowledges that bounds on hidden-layer widths are needed. These hyperparameters are hand-tuned with no sensitivity analysis, and the decreasing-width constraint is a substantial inductive bias that excludes many architectures. The paper should report sensitivity to these choices or reduce the automaticity claim accordingly.","section":"§4.3.3, §4.3.4"}],"minor_comments":[{"comment":"There are numerical inconsistencies: the text in Section 4.2.2 says each sampled network is trained for five epochs, while the caption of Table 1 says three epochs; Section 4.2.2 says 100 configurations are sampled, but Section 4.2.3 says random search picked 200 networks. Please reconcile these numbers.","section":"§4.2.2, Table 1, §4.2.3"},{"comment":"The paper alternates between validation accuracy, test accuracy, and testing accuracy without a consistent definition. For example, Section 4.3.2 Table 7 reports Validation Accuracy, while Section 4.3.4 discusses testing accuracy, and the columns labeled Test Acc in Table 9 are produced by a procedure that Section 4.3.3 describes as validation-based. This makes it hard to audit which data was used for each decision.","section":"§4.2.2, §4.3.2, Table 9"},{"comment":"Figure 9 is described as an animation, but in the PDF it is a static plot; either embed a playable animation or change the description to a static exploration plot.","section":"Figure 9"},{"comment":"There are several typos and small errors, including \"constant form of constant estimation\" (Section 3.1), \"ommit\" (Section 4.3.3), and \"algortihms\" (Section 4.3.4). A careful proofread is recommended.","section":"§3.1, §4.3.3, §4.3.4"},{"comment":"The manual-search baseline is a single network trained without dropout, but the Keras example it is based on uses dropout and reports 98.40%; please state explicitly that this baseline is intended as a no-regularization reference and not as a state-of-the-art comparison.","section":"§4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an undergraduate thesis posted to arXiv, and it reads as a thesis: broad survey, extensive appendix, somewhat informal prose. The central idea is interesting and the experimental documentation is unusually candid, but the headline comparison must be redone with a valid hold-out protocol and the claims should be narrowed or supported by additional benchmarks. I would not reject on novelty alone, but I would not accept until the test-leakage issue is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe thing to know about this paper is that it is a bachelor thesis with one genuinely new idea and one invalid comparison. The idea, automated forward thinking (AFT), combines Hettinger et al.'s greedy layer-wise training with a Cascade-Correlation-style candidate pool. That combination is, as far as I can tell, new, and it is a sensible thing to try. The thesis also adds CaserRe, a small but reasonable twist on Caser that reuses the previous output weight vector. The literature review is solid, the writing is clear, and the author is candid about negative results—Cascor just does not work on MNIST.\n\nThe soft spot is structural. The central claim is that AFT beats every MLP investigated, including random and evolutionary search. But Sections 4.2.2 and 4.2.3 say those baselines were selected using the testing set: random search is assessed on train and test, and evolutionary fitness is test accuracy after three epochs. AFT, by contrast, selects layers on validation accuracy. Comparing a test-selected baseline against a validation-selected construction inflates AFT's apparent edge. On top of that, the headline AFT number—97.86% for [900,600,600,300]—is a best-of-20 choice also made on the test set, so it is not a clean estimate either. The average over the 20 runs is 97.54% with sigma 0.17%, which overlaps with the baselines once their selection bias is considered. So the reported advantage is probably mostly noise.\n\nOther soft spots are proportionate to the format: one dataset (MNIST), feedforward nets only, no released code, and a lot of hand-set hyperparameters without sensitivity analysis. None of these kill the underlying idea, but they do mean the paper cannot support the strong claim it makes. It is an honest, well-scoped thesis, and the AFT algorithm deserves a proper validation-based re-run, ideally on a second dataset, with code released.\n\nWho should read it: anyone working on constructive neural networks or cheap NAS. It is not a finished result, but it is a reasonable starting point and a good example of how easy it is to leak the test set into architecture search. I would not desk-reject it; a serious referee could ask for the clean protocol and a re-run, and the paper could then be a useful workshop contribution. But as it stands, it is a conditional reject / major revision.","headline":"A genuinely new but very preliminary constructive algorithm; the head-to-head MNIST claim collapses once you notice the baselines were selected on the test set, so treat it as an idea worth testing, not a result.","tokens_in":35328,"tokens_out":3001,"would_cite":false,"duration_ms":29509,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated forward thinking builds MNIST classifiers layer by layer from a candidate pool, reaching about 97.5 percent test accuracy—slightly better than every multilayer perceptron the thesis tested.","keywords":["automated architecture design","forward thinking","candidate pool","constructive neural networks","neural architecture search","evolutionary search","MNIST digit classification","dynamic learning"],"falsifier":"Re-run the random and evolutionary searches exactly as described but choose architectures using a held-out validation set, never touching the 10,000-example test set until the final selected model is evaluated; if the best fully trained baselines then reach or exceed the 97.54 percent average of automated forward thinking on the same test split, the claim that automated forward thinking outperforms all tested MLPs is falsified.","tokens_in":34155,"feed_emoji":"🧠","tokens_out":7835,"duration_ms":74906,"temperature":0.7,"pith_summary":"The paper tries to show that neural network architecture design can be automated without sacrificing accuracy. Its central proposal, automated forward thinking, grows a feedforward network one hidden layer at a time and selects each layer from a pool of randomly sized candidates by how well it improves validation accuracy. On the MNIST digit task, twenty runs of this algorithm produced architectures with an average test accuracy of 97.54 percent and a best architecture of four layers—[900, 600, 600, 300]—reaching 97.86 percent. That is slightly above every multilayer perceptron baseline in the thesis, including manual, random, and evolutionary search, and it does so while removing most human architectural decisions. A practical reader should care because it suggests that an expensive, expertise-driven part of deep learning can be replaced by a cheap greedy construction rule.","feed_headline":"Layer-picking algorithm tops all tested networks on MNIST","feed_subtitle":"Automated forward thinking chooses each layer from a candidate pool and reaches 97.5 percent test accuracy.","key_machinery":"The mechanism is a candidate-pool layer selector grafted onto forward thinking. Forward thinking is a greedy training scheme that adds one hidden layer at a time: the new layer is trained on the network's current output, its input weights are frozen, and the output layer is discarded and retrained, so error never has to flow through more than one fresh layer. Automated forward thinking adds a search step: a pool of eight candidate layers, each a random width between 50 and 1000 in multiples of 50, is trained for two epochs, and the candidate with the best validation accuracy is the one inserted. A stricter stopping rule—or equivalently, training extra layers and then picking the layer count with the best accuracy-versus-complexity trade-off—keeps the final network small. This machinery carries the argument because it converts architecture selection from a human hyperparameter choice into a local, greedy, randomized search over layer widths.","core_discovery":"The discovery is an algorithm, named automated forward thinking, that turns the greedy layer-wise training idea of forward thinking into a complete architecture search. Starting from the input, the algorithm keeps a pool of eight candidate hidden layers whose widths are drawn uniformly from 50 to 1000 in steps of 50; each candidate is trained for two epochs, the one with the highest validation accuracy is inserted into the network, its input weights are frozen, the output layer is retrained, and the process repeats. Across 20 runs on MNIST this produced networks with an average test accuracy of 97.54 percent and a standard deviation of 0.17 percent, with the best architecture [900, 600, 600, 300] reaching 97.86 percent. The thesis concludes that automated forward thinking builds deeper networks than the standard two-hidden-layer MLPs found by manual, random, and evolutionary search, and that it yields better test performance than any MLP investigated in the thesis.","pith_inferences":["The candidate-pool layer selector is a general constructive search primitive; the same pool-and-select step could be applied to convolutional filters, recurrent cells, or residual blocks and tested on other data sets, though the thesis only demonstrates feedforward MLPs on MNIST.","The paper's baseline comparisons are biased in favor of random and evolutionary search: both selected architectures using test-set accuracy, so a validation-only rerun could shrink or erase the roughly 0.2-percentage-point margin over the best MLPs.","The post-hoc layer-count selection is essentially a complexity-penalized model selection rule; formalizing it as accepting a layer only when validation loss improves by at least a specified margin would turn the heuristic into a stopping rule with a sharper complexity guarantee.","If the margin over baselines survives a clean validation protocol, the practical consequence is that architecture search for feedforward networks can be run on a single GPU in minutes rather than requiring large parallel search farms."],"forward_implications":["If the central claim holds, deep feedforward networks can be constructed layer by layer with no backpropagation through the full stack, which removes the vanishing-gradient obstacle that usually limits depth.","On MNIST, a fully automated construction rule reaches about 97.5 percent test accuracy—slightly better than the best manually designed or searched multilayer perceptrons—suggesting that human architecture expertise buys little on this task once a good greedy selector is in place.","The small standard deviation of 0.17 percent across runs means the improvement over the 97.3 percent baselines, though modest, is consistent rather than the product of a single lucky run.","Because the selection criterion is validation accuracy, automated forward thinking also supplies a natural early-stopping rule for depth, which can be tuned against model size.","The same layer-by-layer construction can be stopped after any number of layers, so the algorithm produces a family of architectures of increasing depth and lets the user choose the point where added complexity stops paying for itself."],"supporting_citations":[{"why":"Introduces forward thinking, the greedy one-layer-at-a-time training framework that the paper extends with a candidate-pool architecture search.","marker":"[Hettinger et al., 2017]"},{"why":"Supplies the cascade-correlation algorithm whose candidate-unit pool inspired the candidate-layer selection, and provides the Cascor baseline.","marker":"[Fahlman and Lebiere, 1990]"},{"why":"Defines the MNIST digit-recognition benchmark used for all experiments.","marker":"[LeCun et al., 1998]"},{"why":"Establishes random search as a standard hyperparameter-optimization baseline and motivates the random search comparison.","marker":"[Bergstra and Bengio, 2012]"},{"why":"Provides the search-space, search-strategy, and performance-estimation taxonomy that frames the comparison of search algorithms.","marker":"[Elsken et al., 2019]"},{"why":"Proposes the Caser variant of cascade-correlation and supplies one of the cascading baselines the thesis re-implements.","marker":"[Littmann and Ritter, 1992]"},{"why":"Surveys evolving artificial neural networks and supplies the evolutionary architecture-search framework used as an adaptive baseline.","marker":"[Yao, 1999]"},{"why":"Provides the evolutionary algorithm parameter values used in the evolutionary search experiments.","marker":"[Harvey, 2017]"},{"why":"Supplies the example MLP architecture that the manual-search baseline adapts.","marker":"[Keras, 2019]"}],"fun_headline_variants":["Automated forward thinking hits 97.5% on MNIST","Layer-by-layer search beats manual design","Algorithm picks hidden layers iteratively, wins on MNIST","Forward thinking automates deep network architecture","Greedy layer selection yields 97.5% MNIST accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats the reported test accuracies as unbiased measures of generalization, but the random and evolutionary baselines chose architectures using the test set itself, so those baselines can only look better than a clean validation protocol would make them; if that bias is corrected, automated forward thinking's margin may disappear.","fun_headline_variants_meta":{"raw":{"variants":["Automated forward thinking hits 97.5% on MNIST","Layer-by-layer search beats manual design","Algorithm picks hidden layers iteratively, wins on MNIST","Forward thinking automates deep network architecture","Greedy layer selection yields 97.5% MNIST accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1626,"prompt_tokens":907,"completion_tokens":719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":641}},"tokens_in":523,"tokens_out":719,"duration_ms":6950,"temperature":1.0,"reasoning_tokens":641,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:48:20.368554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the random and evolutionary searches exactly as described but choose architectures using a held-out validation set, never touching the 10,000-example test set until the final selected model is evaluated; if the best fully trained baselines then reach or exceed the 97.54 percent average of automated forward thinking on the same test split, the claim that automated forward thinking outperforms all tested MLPs is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cascade-correlation algorithm whose candidate-unit pool inspired the candidate-layer selection, and provides the Cascor baseline."},{"cited_title":"and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"Establishes random search as a standard hyperparameter-optimization baseline and motivates the random search comparison."},{"cited_title":"H., and Hutter, F","cited_arxiv_id":null,"evidence_quote":"Provides the search-space, search-strategy, and performance-estimation taxonomy that frames the comparison of search algorithms."},{"cited_title":"and Ritter, H","cited_arxiv_id":null,"evidence_quote":"Proposes the Caser variant of cascade-correlation and supplies one of the cascading baselines the thesis re-implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys evolving artificial neural networks and supplies the evolutionary architecture-search framework used as an adaptive baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary algorithm parameter values used in the evolutionary search experiments."},{"cited_title":"Simple Deep Neural Network on the MNIST Dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the example MLP architecture that the manual-search baseline adapts."}],"review_version":1}