{"id":"507df8c5-ad78-4840-b799-f0a514a27f52","arxiv_id":"2504.18490","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Adding the CBAM attention module to ResNet50 lowered the reported MAPE for PCI prediction from pavement images to 58.16%, from 70.76% for ResNet50 and 65.48% for DenseNet161.","lead":"This paper tests an image-classification network with an added focus mechanism that estimates a road condition score directly from pavement photos, reporting lower average error than two baseline networks. It is a small benchmark study whose headline numbers lack the error bars, held-out test set, and clean data split needed to back the claim of significant improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmentation-before-split risks train/validation leakage, which could invalidate the reported MAPE gap; the paper never states the split is grouped by source image.","rationale":"The reader's weakest assumption is exactly the most load-bearing concern: the order of augmentation and splitting creates a real risk of train/validation leakage that would invalidate the reported performance comparison. This is not a minor methodological quibble; it is the hinge on which the entire central claim turns. The paper's own description in Section IV.A is unambiguous about applying augmentation before the split, and no statement of source-image grouping appears anywhere. The reader also correctly notes the lack of a held-out test set and the absence of error bars, but the leakage concern is the most direct and decisive because it can produce artificially low validation error for all models, and especially for the model with extra capacity to memorize. I agree with the reader's assessment, and therefore the recommendation remains REJECT (i.e., no change to the reader's verdict).","tokens_in":14387,"tokens_out":1751,"duration_ms":17508,"concrete_test":"Obtain the DSPS dataset and augmentation code; for each of the 7,704 source images, record whether any augmented variant appears in both training and validation sets (e.g., by matching image hashes or file provenance in the split script). If leakage exists, rerun all three models with a source-image grouped split (augment only training images after the 90/10 split) using multiple seeds. If the MAPE gap between ResNet50+CBAM and the baselines disappears or falls within run-to-run variance, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ResNet50+CBAM achieves a 'significantly lower' MAPE than the baselines—rests entirely on validation metrics computed after a 90/10 split described in Section IV.A. The text says all 7,704 images were resized and augmented (color jitter, horizontal/vertical flips) into 30,816 images, and only then 'the dataset was split into training and validation sets.' This order implies the split was performed on the augmented set, not on the original 7,704 source images. If the split was not grouped by source image, augmented variants of the same photograph can appear in both training and validation, allowing the model to memorize source-specific texture and lighting. That would inflate validation scores, and the entire head-to-head comparison between ResNet50, DenseNet161, and ResNet50+CBAM would be comparing on partially seen data. The paper never states that the split was performed per source image or that augmentation was applied only after splitting. This is a concrete, load-bearing flaw: if true, the reported 58.16% vs. 70.76% and 65.48% MAPE numbers are not trustworthy evidence of generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes augmenting ResNet50 with a Convolutional Block Attention Module (CBAM) to predict the Pavement Condition Index (PCI) directly from pavement images, without per-crack annotations. Using the DSPS competition dataset of 7,704 images augmented to 30,816 images, the authors compare ResNet50+CBAM against ResNet50 and DenseNet161 baselines on RMSE, MAE, and MAPE. They report lower errors for the proposed model (e.g., MAPE 58.56% vs 70.76% and 65.48%) and claim that this is the first direct image-to-PCI approach without human-labeled data. The experimental section also includes attention map visualizations and R² scatter plots.","tokens_in":14397,"tokens_out":4782,"duration_ms":44038,"significance":"If the reported improvements are reliable, the paper offers a modest but practically useful checkpoint for automated pavement condition assessment: adding CBAM to a standard CNN gives a simple way to regress PCI from images, with visual interpretability. The use of a public benchmark dataset and the clear task framing are strengths, and the paper explicitly describes its metrics. However, the significance is undercut by load-bearing issues: the training/validation split is described after data augmentation, creating a risk of leakage; the headline \"significant\" improvement rests on single-run metrics with no error bars or significance testing; and the novelty claim is in tension with the authors' own cited prior work (reference [4], Image2PCI). These issues prevent the current results from supporting the paper's central claims.","major_comments":[{"comment":"The dataset is resized and augmented (color jitter, random horizontal and vertical flips) to 30,816 images, and only then \"the dataset was split into training and validation sets\" with 90/10. If the split was made on the augmented set without grouping by source image, augmented variants of the same original photo can appear in both training and validation. Since the reported comparison in Table V (MAPE 58.56 vs 70.76 vs 65.48) is computed on the validation set, the entire head-to-head comparison is compromised under leakage. The paper must state explicitly whether the split was performed on the original 7,704 images (or grouped by source) before augmentation; if not, the experiments must be rerun with a proper split.","section":"Section IV.A"},{"comment":"The abstract and Section V describe the MAPE/MAE differences as \"significantly lower\" or \"significant,\" but the results appear to come from a single training run with no standard deviations, confidence intervals, or significance tests. On the 0–100 PCI scale, the MAE difference between ResNet50+CBAM (13.93) and ResNet50 (14.03) is tiny relative to typical run-to-run variability; MAPE is also sensitive to near-zero PCI values. The authors should report at least three runs with mean±std for each metric and perform a pairwise significance test before using the term \"significant.\" Without this, the central claim of superiority is not statistically supported.","section":"Section V / Abstract"},{"comment":"The claim that \"this is the first effort to associate pavement imagery directly with the PCI without relying on human-labeled data\" is contradicted by reference [4] (Owor et al., Image2PCI), which the authors themselves cite in the Introduction as \"a multitask learning framework for estimating pavement condition indices directly from images.\" Please reconcile this contradiction or revise the novelty claim to be specific about what is actually new relative to that prior work.","section":"Section II.E"},{"comment":"The comparison text states that DenseNet161 has R²=0.56 \"compared to our proposed model with an R² value of 0.56,\" while Section V.A reports R²=0.61 for the proposed model. This internal inconsistency, together with the small differences among R²=0.61, 0.57, and 0.56, does not support the stated conclusion that the proposed model achieves a substantial improvement. The authors should correct the typo and provide uncertainty information for these R² estimates, or the claim of substantial improvement should be tempered.","section":"Section V.B"}],"minor_comments":[{"comment":"The displayed equation for the spatial attention module appears to be a duplicate of the channel attention formula (Eq. 2); it should involve concatenation along the channel axis and a 7×7 convolution, e.g., \\(M_S(F)=\\sigma(f^{7\\times7}([AvgPool(F);MaxPool(F)]))\\).","section":"Section III.C.2, Eq. (3)"},{"comment":"The manuscript header contains boilerplate from a different IEEE submission (MTT-S Microwave Symposium), including author names, addresses, and grant numbers that do not match the current paper; this material should be removed.","section":"Header / Title"},{"comment":"The Acknowledgments section contains a placeholder: \"This should be a simple paragraph before the References to thank those individuals and institutions who have supported your work on this article.\" Please complete or delete this placeholder.","section":"Acknowledgments"},{"comment":"Cross-referencing is inconsistent: the results table is labeled Table V but is referred to as \"Table 3\" and \"Table 1\" in the text (Sections V and V.B). Use consistent table numbering throughout.","section":"Tables / Cross-references"},{"comment":"The description of CBAM insertion is vague (\"CBAM is introduced within the bottleneck\") without specifying the exact positions relative to the residual blocks, the reduction ratio, or whether CBAM is applied to the input or output feature maps of each block. Provide these details for reproducibility.","section":"Section III.D"}],"recommendation":"major_revision","confidential_remarks":"The paper requires additional experimental rigor before it can be considered for publication. The augmentation/split ambiguity in Section IV.A is a concrete, fixable flaw that could invalidate the reported numbers, and the absence of any uncertainty quantification means the headline improvement is not established. The novelty claim should be checked against the authors' own Image2PCI reference. The presentation issues (boilerplate header, incomplete acknowledgments) suggest a rushed submission, but these are minor compared to the experimental concerns. I recommend major revision rather than outright rejection because the proposed architecture is reasonable and the central question could be resolved with properly rerun experiments and statistical reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper reports a real benchmark attempt — ResNet50+CBAM versus ResNet50 and DenseNet161 on the DSPS pavement PCI dataset — but the reported MAPE gap (58.16% vs. 70.76% and 65.48%) is not trustworthy as evidence. The main reason is that the data section describes augmenting all 7,704 images to 30,816 and only then splitting 90/10. If that split is not grouped by source image, augmented copies of the same photo can appear in both training and validation, which would inflate the validation numbers on which every headline result rests. The paper never says the split was done on the original images first.\n\nWhat is genuinely sensible here: the idea of adding CBAM to ResNet50 for direct image-to-PCI regression is reasonable, and comparing against two standard baselines on a public competition dataset is a worthwhile thing to try. The paper also correctly identifies the practical value of avoiding manual distress labeling.\n\nBut the execution is far below submittable standard. The abstract calls the MAPE difference \"significantly lower\" with no error bars, no repeated seeds, and no significance test; the MAE gap is 14.03 vs. 13.93 on a 0–100 scale. There is no held-out test set, only validation with early stopping, so the headline numbers are selected rather than unbiased. The \"first effort\" claim in Section II.E is contradicted by the paper's own references [4] and [23]. The CBAM equations are mis-cited to Zeiler & Fergus, Eq. (3) is the wrong formula (it repeats the channel attention equation), and there are leftover IEEE MTT-S template headers and a placeholder acknowledgments paragraph. The R² values in Section V.B are internally inconsistent.\n\nThis paper is not ready for peer review. The central comparison could become valid if the leak is fixed, the pipeline is corrected, and the paper is rewritten with a proper split, multiple seeds, and an evaluation on a true test set. As it stands, the evidence does not support the claims. I would desk-reject this version and invite a serious resubmission.","headline":"A careless benchmark paper whose headline MAPE improvement is not supported by the reported experiment, largely because augmentation appears to precede the train/validation split.","tokens_in":15194,"tokens_out":2078,"would_cite":false,"duration_ms":20297,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding CBAM attention to ResNet50 lowers pavement PCI prediction error from 70.76% to 58.16% MAPE on a public road-image dataset.","keywords":["Pavement Condition Index","PCI prediction","ResNet50","Convolutional Block Attention Module","CBAM","attention mechanism","image regression","pavement distress"],"falsifier":"Look in the validation set for images that are flips or color-jittered versions of training-set images; if any exist, the reported MAPE gap no longer measures generalization. Re-running the experiment with a per-source-image split and recomputing the three MAPE values would settle whether the CBAM advantage survives a clean split.","tokens_in":13949,"feed_emoji":"🛣️","tokens_out":7313,"duration_ms":68239,"temperature":0.7,"pith_summary":"This paper tries to show that inserting a Convolutional Block Attention Module (CBAM) into a ResNet50 regressor lets a neural network predict the Pavement Condition Index (PCI), a 0-to-100 road-quality score, directly from pavement photographs without separate distress annotations. On a public dataset of 7,704 pavement images from three US cities, the authors report that ResNet50-CBAM reaches a mean absolute percentage error (MAPE) of 58.16%, better than plain ResNet50 (70.76%) and DenseNet161 (65.48%), and the highest coefficient of determination (R² = 0.61). The paper argues that the attention module's sequential channel-then-spatial focus is what drives the improvement, and calls this the first direct image-to-PCI association that needs no human-labeled distress data. The practical stakes are cheaper and less subjective road-condition monitoring for maintenance planning.","feed_headline":"Attention layer cuts pavement-score error to 58%","feed_subtitle":"Adding CBAM to ResNet50 beats two standard baselines on direct image-to-PCI regression, no distress labels required.","key_machinery":"The central object is the Convolutional Block Attention Module (CBAM), a lightweight attention module that computes a channel attention map (two spatial pooling descriptors passed through a shared MLP and a sigmoid) followed by a spatial attention map (channel-wise pooling followed by a 7×7 convolution and a sigmoid). The paper places CBAM within the residual bottleneck blocks of ResNet50, so every stage of feature extraction can re-weight which channels and which image locations matter for the regression. This sequential channel-then-spatial refinement is the mechanism claimed to separate pavement distress from background without explicit annotation.","core_discovery":"The central claim is that adding CBAM inside the bottleneck blocks of ResNet50 improves regression of PCI from raw top-down pavement images. CBAM first recalibrates channels via average- and max-pooling through a shared MLP, then recalibrates spatial locations via a 7×7 convolution; the paper reports that this feature-refinement step lowers MAPE from 70.76% and 65.48% to 58.16%, with the smallest RMSE and MAE of the three models, and raises R² to 0.61. In the authors' reading, the improvement comes from targeted feature refinement rather than added capacity, because the parameter increase over ResNet50 is small. They also assert that this is the first effort to relate pavement imagery directly to PCI without relying on human-labeled data, and show attention maps that highlight cracks and potholes as evidence the model learns distress-like regions on its own.","pith_inferences":["My inference: because MAPE divides by true PCI, the reported gap may be carried mostly by low-PCI images; reporting error per PCI band would show whether CBAM genuinely improves the cases most relevant to maintenance.","My inference: the 'first image-to-PCI without human labels' claim is best read as first for a CBAM-style self-attention regressor trained only on image-to-PCI pairs, since direct image-to-PCI regression already exists in the literature the paper cites.","My inference: a decisive and cheap extension is to ablate channel and spatial attention separately and to re-split the data by source image; if the CBAM advantage disappears under either change, the headline MAPE gap is not robust."],"forward_implications":["If the result holds, transportation agencies can estimate PCI for a road network directly from routine camera images, skipping the manual distress surveys that make current PCI assessment slow and subjective.","Attention-based feature refinement, not raw parameter count, becomes the credible explanation for the accuracy gain: CBAM adds few parameters yet produces the largest error reduction, while the heavier DenseNet161 gains little over ResNet50.","The model is most reliable on well-maintained pavements (high PCI) and least reliable on heavily degraded ones, so immediate deployment would be safest for screening roads that are likely in good condition, with low-PCI cases flagged for manual review.","A physics-informed loss function that encodes how pavements deteriorate over time is a plausible next step the paper proposes to improve low-PCI predictions and generalization to unseen conditions."],"supporting_citations":[{"why":"Supplies the ResNet50 residual architecture the paper modifies; the baseline and backbone for the proposed CBAM-augmented model.","marker":"[47]"},{"why":"Provides the attention-mechanism survey that motivates channel and spatial attention and the expected gain from focusing on salient features.","marker":"[24]"},{"why":"Earlier attention-based pavement crack detection work the paper draws on for the idea that attention helps identify distress features; the proposed model extends this to direct PCI regression.","marker":"[26]"},{"why":"Describes a multitask framework estimating PCI directly from images, giving the closest existing approach against which the paper's novelty and comparison are positioned.","marker":"[4]"},{"why":"Documents the pavements-data challenge from which the 7,704 labeled pavement images are drawn, establishing the dataset used for training and validation.","marker":"[15]"}],"fun_headline_variants":["Attention module cuts pavement PCI error to 58%","ResNet50+CBAM predicts PCI straight from photos","CBAM helps ResNet50 spot cracks, beat baselines","Direct PCI prediction: attention beats two CNNs","MAPE drops to 58.16% with attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 90/10 train/validation split happened before augmentation, so no augmented copy of a source pavement photo appears in both sets; if the split was made after augmentation without grouping by source image, the reported validation metrics are inflated and the head-to-head comparison is not a clean generalization test.","fun_headline_variants_meta":{"raw":{"variants":["Attention module cuts pavement PCI error to 58%","ResNet50+CBAM predicts PCI straight from photos","CBAM helps ResNet50 spot cracks, beat baselines","Direct PCI prediction: attention beats two CNNs","MAPE drops to 58.16% with attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3294,"prompt_tokens":903,"completion_tokens":2391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2312}},"tokens_in":519,"tokens_out":2391,"duration_ms":17467,"temperature":1.0,"reasoning_tokens":2312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:16:39.547175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look in the validation set for images that are flips or color-jittered versions of training-set images; if any exist, the reported MAPE gap no longer measures generalization. Re-running the experiment with a per-source-image split and recomputing the three MAPE values would settle whether the CBAM advantage survives a clean split.","supporting_citations":[{"cited_title":"Image2PCI -- A Multitask Learning Framework for Estimating Pavement Condition Indices Directly from Images","cited_arxiv_id":"2310.08538","evidence_quote":"Describes a multitask framework estimating PCI directly from images, giving the closest existing approach against which the paper's novelty and comparison are positioned."},{"cited_title":"The 1st Data Science for Pavements Challenge","cited_arxiv_id":"2206.04874","evidence_quote":"Documents the pavements-data challenge from which the 7,704 labeled pavement images are drawn, establishing the dataset used for training and validation."}],"review_version":1}