{"id":"535e902e-274d-454a-9dbf-5fdce9a448a5","arxiv_id":"2411.17140","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"The proposed Attention-ResNet50-GA pipeline reports 0.9967 precision and 0.9983 F1 for crack detection, but missing evaluation details prevent verification.","lead":"This paper combines a pretrained ResNet50 image model, a spatial attention layer, and a genetic algorithm that tunes the final network layers to detect cracks in road and concrete photos. The reported accuracy is near perfect, but the paper does not release code or describe its data split, so the result cannot be independently checked.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported near-perfect precision/F1 is load-bearing on a held-out test split, but the paper never defines one; the GA fitness is 'validation accuracy,' so final metrics may be self-selected.","rationale":"I read the paper as an applied incremental claim: standard pretrained ResNet50 + CBAM-style spatial attention + a small dense head evolved by GA produces near-perfect crack classification on a chosen dataset. For that claim to hold, the reported metrics must be an honest estimate of performance on data not used to make decisions. The weakest point is the evaluation protocol. The paper states the GA fitness is validation accuracy and never states a train/validation/test split; the data section describes a manually selected 6,000-image subset but no partition. This is exactly the leak that would make a 0.9983 F1 uninformative. The reader's weakest assumption identifies the same issue, so I agree. I do not see an internal contradiction in the spatial attention equations, and the parameter-count increase of about 0.44M for the GA head is plausible for the reported layer sizes. The problem is not that the components are wrong; it is that the empirical result is not verifiable from the manuscript. The limitation paragraph in the Conclusion discusses compute but not the missing split, so the omission is not acknowledged. One concrete check—release the split and rerun the final architecture on a truly held-out test set—would settle whether the central claim survives. Until then the appropriate verdict is REJECT, keeping the reader's verdict unchanged.","tokens_in":9277,"tokens_out":4978,"duration_ms":44048,"concrete_test":"Obtain the exact train/validation/test partition and the full list of images used in the manual 6,000-image subset. Then, fixing the GA-selected architecture (4 layers; 66, 805, 218, 382 neurons), retrain only on the declared training split and compute precision/recall/F1 on a previously untouched test split (e.g., 1,000 balanced images) that was never used for GA fitness, early stopping, or subset selection. If no such test split exists, or if the test F1 falls materially below 0.9983 (or is not reproducible with released code/data), then the near-perfect claim is unsupported by the paper as written.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that Attention-ResNet50-GA achieves 0.9967 precision and 0.9983 F1 on crack images—requires that these numbers come from images never used to select the model or its data subset. The paper never states such a split. In the 'Genetic Algorithm Process / Fitness Function' subsection, the fitness is explicitly f(individual) = validation accuracy, and Figure 4 shows training/validation curves, not test curves. The 'Data Acquisitions and Pre-processing' section says a balanced subset of 6,000 images was manually selected from 'various sources' to mitigate homogeneity, but it does not say how those images were chosen or whether they were separated before subsetting. If Table 1 metrics are computed on the same validation fold that drove GA architecture selection—or on images that influenced the manual subset choice—then the GA has effectively selected the validation set, and the near-perfect numbers are an artifact of selection rather than evidence of generalization. The paper also provides no code, data, seeds, or repeated runs, so the result cannot be checked, and no confidence interval is available. This is not a disagreement with the standard spatial-attention/transfer-learning machinery; it is a minimum-validity condition for the empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a pipeline for binary crack detection in infrastructure images: a frozen ImageNet-pretrained ResNet50 is used as a feature extractor, the features pass through a spatial attention module, and the resulting representation is classified by a small fully connected network whose layer count and layer widths are optimized by a genetic algorithm. The authors evaluate four configurations (a CNN trained from scratch, plain ResNet50 transfer learning, attention-augmented ResNet50, and attention-augmented ResNet50 with a GA-optimized head) on a balanced subset of 6,000 images drawn from the CFD and CCICD datasets. Table 1 reports precision 0.9967, recall 1.0, and F1 0.9983 for the proposed Attention-ResNet50-GA model, which the authors interpret as near-perfect and superior crack detection. The paper does not describe an independent test set, repeated runs, error bars, or code/data availability.","tokens_in":9535,"tokens_out":8358,"duration_ms":73608,"significance":"The proposed combination of frozen pretrained features, spatial attention, and evolutionary search over the classifier-head architecture is a reasonable and clearly described idea, and the paper documents the GA representation, crossover, and mutation operators in more detail than is common. If the reported numbers were measured on a held-out test set, the result would be a useful incremental demonstration, though not a conceptual surprise, since the components are all established. The central empirical claim is not currently supported, however: the manuscript never defines a test set, the GA fitness is explicitly validation accuracy, and no repeated runs or error bars are reported. The absence of code and data also prevents independent checking. Consequently, the paper's main quantitative contribution—the near-perfect precision and F1—cannot be verified or interpreted as evidence of generalization.","major_comments":[{"comment":"The central claim is load-bearing on an independent test set, but the manuscript never describes one. In the Fitness Function subsection, f(individual) is defined as validation accuracy, and Figure 4 shows training and validation curves only. If the numbers in Table 1 were computed on the same validation set that guided the GA, then the high precision and F1 are partially artifacts of selection rather than evidence of generalization. The authors must specify the train/validation/test split, state that the reported metrics are from a test set never used in GA fitness evaluation or model selection, and give the sizes of the three partitions.","section":"Case Study / Result (Table 1), with 'Fitness Function' in 'Optimization Using Genetic Algorithm'"},{"comment":"The construction of the evaluation dataset is not described with enough precision to assess bias. The paper states that a balanced subset of 6,000 images (3,000 per class) was selected from CFD and CCICD 'to mitigate' the homogeneity of CCICD, but it does not state the selection criterion, whether the selection was random or manual, or how the subset was divided among the two sources. If the subset was chosen with knowledge of the labels or after looking at the images, the reported performance can reflect dataset selection rather than model quality. The authors should describe a reproducible sampling procedure and perform the subset selection before any test split, without access to test labels.","section":"Data Acquisitions and Pre-processing"},{"comment":"All quantitative claims rest on single-run point estimates. The GA is stochastic—initialization, roulette-wheel selection, random pairing, and mutation all involve randomness—and the metric differences are small on recall (0.9797 vs 1.0). Without multiple seeds, confidence intervals, or a statistical comparison, the conclusion that Attention-ResNet50-GA 'outperforms' the baselines is not established. Please report the mean and standard deviation over repeated runs and, if possible, a paired significance test across the same folds.","section":"Result (Table 1) and 'Genetic Algorithm Process'"},{"comment":"The GA is a core component, yet the manuscript does not report the values of its key hyperparameters: population size N, mutation probabilities p_add, p_remove, and p_neuron, the allowed neuron range [n_min, n_max], the number of generations, and the convergence threshold. Only the maximum number of layers (5) is stated. These values are necessary both for reproducibility and for judging whether the search was sufficient to support the optimality claim.","section":"Optimization Using Genetic Algorithm"}],"minor_comments":[{"comment":"The abstract contains a typo: 'inaccessability' should be 'inaccessibility'.","section":"Abstract"},{"comment":"The notation P1 = {L(1)_1, ..., L(1)_n1} is confusing because L is used both for the number of layers and for the layer sizes; please use distinct symbols, e.g., l_i for the number of neurons in layer i.","section":"Genetic Algorithm Process / Crossover"},{"comment":"The phrase 'with 66 , 805, 218, 382 being the number of the neurons in each layer' is ambiguous; write the architecture as [66, 805, 218, 382] or use a clearer delimiter so the per-layer sizes are unambiguous.","section":"Result"},{"comment":"Please add a data availability statement or specify where the code and the exact subset of images can be obtained; currently the experiments cannot be reproduced.","section":"Data Acquisitions and Pre-processing"}],"recommendation":"reject","confidential_remarks":"The paper is not ready for publication. The evaluation-protocol gap alone invalidates the headline result, and the absence of repeated runs and code means that a full new experimental study would be needed before the empirical claim could be assessed. I would recommend rejection rather than a routine revision, because the numbers as reported cannot be interpreted as evidence of generalization. If the authors later provide a proper train/validation/test protocol, repeated seeded runs, and reproducible data/code, the underlying idea could be worth revisiting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an incremental applied paper that stacks three off-the-shelf ideas — frozen ImageNet ResNet50 features, a CBAM-style spatial attention block, and a GA-searched classifier head — for infrastructure crack detection. The actual novelty is the combination itself, and the empirical centerpiece — precision 0.9967, F1 0.9983 — cannot be evaluated as reported. The paper never states a train/validation/test split. The GA fitness function is explicitly validation accuracy, and no independent test set is described anywhere. If Table 1 comes from the same validation set used for architecture selection, the near-perfect scores are partly self-selected. That is a load-bearing omission, not a style quibble.\n\nCredit where it is due: the spatial attention equations are standard and correctly laid out; the GA encoding, crossover, and mutation are described in enough detail to reimplement; and the limitation section honestly acknowledges that attention alone did not help and that GA is computationally expensive. The choice of ResNet50 because the cited comparison study did not cover it is reasonable and clearly motivated. The literature review is adequate for an applied paper.\n\nSoft spots, in proportion. The missing split is the big one. There are also no seeds, repeated runs, confidence intervals, code, or data release, so the near-perfect result cannot be checked. GA hyperparameters (population size, mutation probabilities, neuron range, generations) are never given. The 6,000-image subset was 'manually selected' from various sources, but the selection criteria and whether selection happened before any split are unclear. On the modeling side, the spatial attention block is essentially CBAM without the channel branch, yet there is no comparison against a standard CBAM-ResNet50 baseline, which would be the natural control. It is also odd that adding attention alone lowers precision (0.914 vs 0.9667) while GA plus attention jumps to 0.9967; the paper's sample-size explanation is hand-wavy.\n\nCitation pattern: fine. The references to transfer learning, attention, and GA in crack detection are appropriate, and the paper does not oversell the literature.\n\nBottom line: desk reject in current form. The author should release code and data, define the split, report repeated runs with error bars, and add the CBAM baseline. If those are supplied, the result might be worth a second look. As it stands, I would not spend referee time on it.","headline":"Incremental CBAM-style attention plus GA-tuned head; the near-perfect numbers are unverifiable because no test split is ever defined, and the GA selects on validation accuracy.","tokens_in":10050,"tokens_out":3436,"would_cite":false,"duration_ms":30227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a frozen ResNet50, a spatial attention layer, and a genetic-algorithm-tuned classifier detect cracks with F1 score 0.9983.","keywords":["crack detection","transfer learning","ResNet50","spatial attention","genetic algorithm","structural health monitoring","image classification","hyperparameter optimization"],"falsifier":"Run the proposed Attention-ResNet50-GA pipeline on a fixed, pre-registered split that holds out, say, 20% of the 6,000 images before any GA run, training the GA only on the training fold with its own validation fold, and report metrics on the untouched test fold; if the F1 score drops substantially from 0.9983, the near-perfect result is an artifact of selection on the validation set. Alternatively, apply the trained model to the full CCICD test set or to the 329 Crack Forest images and check whether precision and recall remain at the reported level.","tokens_in":9010,"feed_emoji":"🏗️","tokens_out":9816,"duration_ms":72952,"temperature":0.7,"pith_summary":"The paper sets out to show that crack detection in infrastructure images can be made almost flawless even when labeled data is scarce, by chaining three ideas: frozen ImageNet features from ResNet50, a spatial attention layer that highlights crack-like regions, and a small custom classifier whose layer sizes are selected by a genetic algorithm. On a balanced 6,000-image subset drawn from the Crack Forest and Concrete Crack Images for Classification datasets, the resulting Attention-ResNet50-GA model reports precision 0.9967, recall 1.0, and F1 score 0.9983, surpassing a CNN trained from scratch, plain transfer-learned ResNet50, and ResNet50 with attention only. The practical stake is that a high-accuracy, low-data pipeline could reduce reliance on expensive manual inspection and large annotated datasets.","feed_headline":"Genetic-tuned attention model detects cracks at 99.8% F1","feed_subtitle":"Attention plus genetic-algorithm tuning beats CNN and plain ResNet baselines on crack images.","key_machinery":"The load-bearing mechanism is the three-stage pipeline. First, ResNet50 pretrained on ImageNet is used with frozen weights as a fixed feature extractor. Second, a spatial attention module computes average-pooled and max-pooled channel maps, concatenates them, applies a convolution and sigmoid to produce a per-pixel weight map, and multiplies it back into the feature tensor. Third, a customized fully connected network—whose number of layers (chosen as 4) and neurons per layer (66, 805, 218, 382) are encoded as a chromosome and evolved by a genetic algorithm using roulette-wheel selection, random pairing, crossover, and mutation, with validation accuracy as fitness—turns the attended features into a crack/no-crack decision.","core_discovery":"The paper's central claim is that the Attention-ResNet50-GA model—a ResNet50 feature extractor kept frozen, a spatial attention map built from channel-wise average and max pooling followed by a sigmoid-activated convolution, and a four-layer fully connected head with 66, 805, 218, and 382 neurons found by the genetic algorithm—detects cracks with near-perfect accuracy. The reported numbers are precision 0.9967, recall 1.0, and F1 score 0.9983 on the described dataset, outperforming all three baselines in Table 1. The paper attributes the gain to the GA-selected head, since the attention-only variant (Att-ResNet50) actually had lower precision than plain transfer-learned ResNet50, showing that attention alone does not help without the customized, optimized classifier.","pith_inferences":["One testable extension is to compare the GA-selected head against a random search over the same layer-count and neuron ranges; if random search matches the performance, the gain comes from added capacity rather than from the evolutionary search itself.","Because the GA fitness function is validation accuracy and no test split is described, the near-perfect numbers could reflect selection on the validation set; re-evaluating on a fixed, untouched test split, or on the full CCICD test set, would show whether the claim holds out of sample.","The same pipeline could be evaluated per surface material (asphalt, concrete, masonry) to see whether the attention layer localizes cracks differently across conditions, since the paper's mixed dataset does not separate them.","Once trained, the spatial attention map could be exported as a coarse crack-localization heatmap, giving free localization information alongside the classification label."],"forward_implications":["If the reported metrics reproduce on a properly held-out test set, near-perfect crack detection is achievable with only a few thousand labeled images because the pretrained feature extractor supplies most of the representation.","The comparison implies that adding a spatial attention layer to a frozen backbone can hurt precision when trained on a small set, but that a GA-tuned classification head can more than compensate.","A deployable version could reuse the same frozen ResNet50 and attention layer for new inspection surfaces by re-running the GA search on a small domain-specific sample.","The GA-selected architecture of 4 layers with 66, 805, 218, and 382 neurons gives practitioners a concrete starting point without repeating the search.","The reported model size of 91.69 MB and 24.03 million parameters is modest enough for edge deployment if the frozen backbone is kept intact."],"supporting_citations":[{"why":"Supplies the Crack Forest Dataset of urban concrete road crack images used in the mixed training set.","marker":"[26]"},{"why":"Supplies the Concrete Crack Images for Classification dataset, the main source of the balanced 6,000-image subset.","marker":"[27]"},{"why":"Motivates the choice of ResNet50 by reviewing pretrained models not covered in prior comparisons.","marker":"[4]"},{"why":"Provides the transfer-learning rationale for reusing ImageNet-pretrained features on small domain datasets.","marker":"[5]"},{"why":"Introduces the attention-mechanism approach to defect detection that the spatial attention layer builds on.","marker":"[6]"},{"why":"Establishes the difficulty of manual architecture tuning that motivates the genetic algorithm.","marker":"[7]"},{"why":"Supports the use of genetic algorithms for hyperparameter tuning in machine learning models.","marker":"[24]"},{"why":"Supports the use of genetic algorithms for optimizing neural network architectures.","marker":"[25]"}],"fun_headline_variants":["GA-optimized attention model hits 99.8% F1 on crack detection","Attention + genetic algorithm boosts crack detection to near-perfect accuracy","ResNet50 with GA-tuned head detects cracks at 99.8% F1","Genetic algorithm tunes attention model for crack detection accuracy","Crack detection: GA-tuned attention beats baselines at 99.8% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported near-perfect metrics assume the final numbers come from a held-out test set that is disjoint from the validation set used to guide the genetic algorithm and from the images that influenced the balanced-subset selection, but no such split is described.","fun_headline_variants_meta":{"raw":{"variants":["GA-optimized attention model hits 99.8% F1 on crack detection","Attention + genetic algorithm boosts crack detection to near-perfect accuracy","ResNet50 with GA-tuned head detects cracks at 99.8% F1","Genetic algorithm tunes attention model for crack detection accuracy","Crack detection: GA-tuned attention beats baselines at 99.8% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2583,"prompt_tokens":915,"completion_tokens":1668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1567}},"tokens_in":531,"tokens_out":1668,"duration_ms":9552,"temperature":1.0,"reasoning_tokens":1567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:28:37.981666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed Attention-ResNet50-GA pipeline on a fixed, pre-registered split that holds out, say, 20% of the 6,000 images before any GA run, training the GA only on the training fold with its own validation fold, and report metrics on the untouched test fold; if the F1 score drops substantially from 0.9983, the near-perfect result is an artifact of selection on the validation set. Alternatively, apply the trained model to the full CCICD test set or to the 329 Crack Forest images and check whether precision and recall remain at the reported level.","supporting_citations":[{"cited_title":"(2016) Automatic road crack detection using random structured forests","cited_arxiv_id":null,"evidence_quote":"Supplies the Crack Forest Dataset of urban concrete road crack images used in the mixed training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Concrete Crack Images for Classification dataset, the main source of the balanced 6,000-image subset."},{"cited_title":"(2023) Deep learning for crack detection on masonry fa¸ cades using limited data and transfer learning","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of ResNet50 by reviewing pretrained models not covered in prior comparisons."},{"cited_title":"J., and Faria, D","cited_arxiv_id":null,"evidence_quote":"Provides the transfer-learning rationale for reusing ImageNet-pretrained features on small domain datasets."},{"cited_title":"(2021) Attention U-Net with Feature Fusion Module for Robust Defect Detection","cited_arxiv_id":null,"evidence_quote":"Introduces the attention-mechanism approach to defect detection that the spatial attention layer builds on."},{"cited_title":"H., and Hutter, F","cited_arxiv_id":null,"evidence_quote":"Establishes the difficulty of manual architecture tuning that motivates the genetic algorithm."},{"cited_title":"D., Bello-Salau, H., Umoh, I","cited_arxiv_id":null,"evidence_quote":"Supports the use of genetic algorithms for hyperparameter tuning in machine learning models."},{"cited_title":"(2022) Deep Learning of Neural Networks Using Genetic Algorithms","cited_arxiv_id":null,"evidence_quote":"Supports the use of genetic algorithms for optimizing neural network architectures."}],"review_version":1}