{"id":"50c5b6a7-82a0-4e8b-ae6f-24892276b905","arxiv_id":"2505.06694","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"NAS-DETR uses a maximum-entropy zero-shot neural architecture search to build a CNN-Transformer backbone for a Deformable-DETR detector, reporting improved sonar object detection on URPC2021 and URPC2022.","lead":"The authors propose NAS-DETR, a sonar-image object detector that combines a zero-shot neural architecture search over a CNN-Transformer backbone with a Deformable-DETR-style decoder. It reports state-of-the-art results on two sonar detection benchmarks, but the supporting analysis has several inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"URPC2022 tables contradict each other on which NAS-DETR configuration is best, and no error bars are reported, so the claimed SOTA margins are not empirically established.","rationale":"The paper's central claim is the stated SOTA performance, so the most load-bearing requirement is that the reported numbers be internally consistent and reproducible. They are not: Table 6 and Section 4.3 credit 0.492 mmAP on URPC2022 to NAS-DETR(A2), while Table 9 gives 0.492 to NAS-DETR(A1) and 0.488 to A2. This reversal undermines any claim about which searched configuration is best and raises doubt about the entire benchmark. The absence of error bars or seed counts makes the 1.3-1.6 point improvements over RT-DETR unverifiable; many detection papers on datasets of this size show run-to-run variation in this range. The reader's weakest_assumption concerned the zero-shot fitness proxy; that is also serious, since Eq. (29) is never correlated with trained accuracy and the variance propagation in Eqs. (15)-(21) ignores the softmax nonlinearity and the dependence of attention weights on the value features. But that concern attacks the explanation of why NAS-DETR works, whereas the table contradiction attacks whether the reported result exists as stated. Both point to rejection; I recommend keeping the reader's verdict unchanged.","tokens_in":20754,"tokens_out":8331,"duration_ms":81102,"concrete_test":"Rerun URPC2022 with NAS-DETR(A1) and NAS-DETR(A2) under identical data splits and training recipe, using at least three random seeds, and report mean ± std for mmAP, mAP50, and mAP75, together with the same repeated runs for RT-DETR. If the A1/A2 ordering and the margin over RT-DETR do not reproduce both Table 6 and Table 9, the SOTA claim is not established; the rerun also resolves whether the discrepancy is a label swap or ordinary noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 and Table 6 report URPC2022 mmAP 0.492 for NAS-DETR(A2) and use that value for the SOTA claim over RT-DETR's 0.476. The ablation table (Table 9) reports the opposite ranking: NAS-DETR(A1) reaches 0.492 mmAP while NAS-DETR(A2) reaches only 0.488. Since A1 and A2 are the two entropy-weight settings of the proposed zero-shot search, the contradiction matters: the paper's account of which searched architecture produced the headline result is internally inconsistent. Moreover, all metrics in Tables 4-9 appear to come from single runs, with no standard deviations, seed counts, or statistical tests. The advertised gains over RT-DETR are 1.3 mmAP points on URPC2021 and 1.6 on URPC2022; without repeated seeds, those margins cannot be distinguished from training noise on datasets of 6,000-9,200 images. The central empirical claim is exactly the SOTA numbers, so this baseline reproducibility issue is load-bearing. A separate concern noted by the reader is that the zero-shot fitness function in Eq. (29) is never validated against trained detection accuracy, and Eqs. (15)-(21) treat softmax attention weights as independent of values; both would need repair, but the contradictory tables alone are sufficient to block acceptance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NAS-DETR, a DETR-style sonar object detector whose CNN-Transformer backbone is selected by a zero-shot, maximum-entropy neural architecture search. Candidate backbones are scored with a differential-entropy fitness function, then combined with a feature pyramid network and a deformable-attention decoder that uses content-position decoupled query initialization, denoising training, and a hybrid loss. Experiments on URPC2021 and URPC2022 report state-of-the-art mmAP of 0.538 and 0.492 at 71.2-73.8 FPS, together with an ablation study and a Spearman correlation analysis between architecture parameters and the entropy-based fitness score.","tokens_in":21018,"tokens_out":6687,"duration_ms":62695,"significance":"If the results hold, the paper addresses an important applied problem and the zero-shot search would be practically valuable because it avoids training during architecture search. The motivation is clear, the experimental comparison is broad, and the idea of searching a CNN-Transformer backbone for low-resolution, noisy sonar images is reasonable. However, the empirical support is currently insufficient: the central SOTA numbers come from single runs with no variance information, one headline result is internally contradictory across tables, the zero-shot fitness is never validated against trained detection accuracy, and the theoretical variance-propagation derivation contains unjustified equalities. The paper also does not provide code or trained models, so the tables are the only evidence for the claims.","major_comments":[{"comment":"The headline URPC2022 result is internally inconsistent. Section 4.3 and Table 6 report mmAP 0.492 for NAS-DETR(A2) and use it as the SOTA comparison against RT-DETR (0.476), while Table 9 reports NAS-DETR(A1) at 0.492 and NAS-DETR(A2) at 0.488. Since A1 and A2 are the two entropy-weight configurations of the proposed search, the paper does not specify which searched architecture produced the headline result. This contradiction prevents the reader from reproducing the central empirical claim and must be resolved by correcting the tables or rerunning the experiments.","section":"§4.3 (Table 6) vs. §4.4 (Table 9)"},{"comment":"All detection metrics are reported as single values with no standard deviations, number of seeds, or statistical tests. The claimed advantages over RT-DETR are 1.3 mmAP points on URPC2021 and 1.6 mmAP points on URPC2022; on test sets of roughly 800-1000 images, these margins are within the range of typical training randomness for DETR-style detectors. The central SOTA claim is therefore not empirically established; at minimum, three to five independent seeds with reported means and variances, or a significance test, are needed.","section":"Tables 4-10"},{"comment":"The variance-propagation derivation for the Transformer block is not valid as written. Equation (18) applies Lemma 2 to attention weights a_ij that are stated to lie in (0,1), yet the text asserts E(a_ij)=0, which is impossible for positive values; the independence of attention weights from value features and the treatment of softmax as a negligible scaling are asserted without justification. Equation (21) is dimensionally inconsistent: a product of d_model, d_ff, and S times the input variance cannot equal the variance of y_ij. Equation (22) then multiplies a variance by a sum of logarithms as if it were an entropy. This undercuts the theoretical basis of the fitness function in Eq. (29).","section":"§3.1, Eqs. (15)-(22)"},{"comment":"The zero-shot fitness function Z(G) is never validated against trained detection accuracy. The paper shows that the selected backbones perform well, but it does not show that architectures with higher Z(G) actually yield higher mAP after training, nor does it compare Z(G) against random backbone selection. In addition, the ablation in Tables 8-9 changes the decoder, query initialization, and loss function simultaneously with the backbone, so the 1.2-1.4% gains cannot be attributed to the NAS search alone. An isolated backbone-only comparison, or a correlation between Z(G) and trained mAP across a sample of backbones, is required to support the search claim.","section":"§3.2 Eq. (29) and §4.4 Tables 8-9"},{"comment":"The correlation analysis is largely tautological. Since the fitness Z(G) in Eq. (29) is a weighted sum of log(var(h^D)) + log(C_in), and the derivation in Eqs. (13)-(22) makes the score increase with depth and width by construction, the strong Spearman correlations between depth and score and between channel count and score are expected from the definition rather than empirical discoveries about sonar architectures. Table 11 also contains an internal inconsistency: the text describes feedforward dimension as showing a significant negative correlation with a reported value of ρ=0.904, but 0.904 is positive. The interpretability claim should be reframed and the numerical reporting corrected.","section":"§4.6.3, Eq. (29), Table 11"}],"minor_comments":[{"comment":"There are multiple typos, including 'Univesity' in the affiliations, 'Porposed' in the section title, 'NAS-SDETR' in Sections 1 and 3, and 'the the' in Section 4.3; these should be corrected.","section":"Throughout"},{"comment":"Several numeric entries are concatenated without separators (for example, '0.8870.9050.9590.95' in Table 5 and similar entries in Table 7), making the category-wise mAP50 values unreadable.","section":"Tables 5 and 7"},{"comment":"The table captions and the text are inconsistent: Table 11 is captioned 'under A1' but reports rows for both Score(A1) and Score(A2), and Table 12 is captioned 'under A2' but similarly mixes settings. The abbreviations 'hdn' and 'btn' are used without definition.","section":"Tables 11-12"},{"comment":"The text references 'Table??' instead of a real table number, and the sampling description in Eq. (39) is incomplete because the text does not define how the differential entropy value E_total is computed for the paired samples.","section":"§4.6.4"},{"comment":"The TensorRT quantization results (294.9 and 288 FPS) are reported without any quantization setup, precision, or reproducibility details, so the deployment claim cannot be assessed.","section":"§4.5"},{"comment":"Reference [43] is cited for the differential-entropy upper bound of a neural network, but the cited paper is about fast FFT-based convolution performance; this appears to be a citation error and should be replaced with the correct source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready in its current form. The table contradiction, missing variance information, and unvalidated zero-shot proxy are load-bearing issues that require additional experiments or substantial corrections. If the authors can resolve these points, the applied contribution may be salvageable, but the current evidence does not support the SOTA claim. I would also encourage the editor to verify the novelty claim of being the first DETR+NAS sonar detector against recent literature, since the related-work discussion is limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper applies zero-shot NAS to a DETR-style detector for sonar imagery. The idea is reasonable and the search space is a genuine extension, but the empirical core is shaky. The two headline tables for URPC2022 contradict each other on which configuration is best, and nothing has repeated-seed variance. I would not stake a claim on these numbers.\n\nWhat is genuinely new: the CNN-Transformer search space and the numerical scale normalization in the entropy estimation are sensible engineering contributions, and the paper does run a broad set of baselines. The TensorRT efficiency numbers are a nice practical addition. The correlation analysis is a good idea in principle—tying architectural parameters to the fitness score would help interpretability.\n\nThe problems. Table 6 reports NAS-DETR(A2) at 0.492 mmAP on URPC2022 and uses that for the SOTA claim over RT-DETR's 0.476; Table 9, the ablation, gives NAS-DETR(A1) 0.492 and NAS-DETR(A2) 0.488. That is a direct contradiction about which entropy-weight setting produces the best architecture. It undermines the headline result. Additionally, no error bars, no seed counts, no significance tests appear anywhere. With datasets of 6000 and 9200 images, a 1.3–1.6 point gap against RT-DETR could easily be training noise.\n\nThe theory has soft spots too. Eq. (4) asserts an equivalence without proof, and the variance propagation through the Transformer treats softmax attention weights as independent of the value vectors and ignores the softmax's nonlinearity. That is not a rigorous foundation for the search objective.\n\nWorst, Section 4.6's correlation analysis is largely tautological. The fitness score in Eq. (29) is explicitly a sum over layers of log terms involving channel count and kernel size, so depth and width are correlated with the score by construction. Reporting a Spearman rho of 0.9 between depth and score as a discovery is not meaningful.\n\nThe ablation also conflates the searched backbone with the decoder and loss changes—RT-DETR with a different backbone is not an isolated test of the NAS.\n\nWho is this for? A reader working on sonar detection might try the architecture, but they would be adopting a configuration that the paper itself is inconsistent about. The paper is not ready for publication. The authors might be able to repair it with repeated seeds, fixed tables, and a more careful derivation, but as is it does not deserve a serious referee.","headline":"A sensible engineering attempt undermined by contradictory tables and missing error bars—the claimed SOTA sonar detection results are not currently supported.","tokens_in":21599,"tokens_out":4630,"would_cite":false,"duration_ms":43127,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NAS-DETR claims that a training-free, entropy-based search — not hand-tuned design — picks the backbone that makes its sonar detector the best on two underwater benchmarks while staying real-time.","keywords":["underwater object detection","sonar imagery","detection transformer (DETR)","zero-shot neural architecture search","maximum entropy","differential entropy","evolutionary algorithm","CNN-transformer hybrid backbone"],"falsifier":"Train ten to twenty backbone architectures sampled across the full range of fitness scores $Z(G)$ inside the otherwise identical detector, and compute the rank correlation between $Z(G)$ and the resulting mean average precision; if the correlation is weak or negative, the entropy proxy is miscalibrated and the reported gains come from the detector components or the FLOPs prior rather than from the search. A second, targeted check would measure the actual variance of the transformer block output at initialization and compare it with the paper's formula $\\sigma^2(y) \\approx d_{\\mathrm{model}}^2\\, d_{\\mathrm{feedforward}}\\, S^2\\, \\sigma^2(x)$; a large mismatch would confirm that treating softmax attention as a linear scaling matrix breaks the derivation.","tokens_in":20496,"feed_emoji":"🎯","tokens_out":19646,"duration_ms":151426,"temperature":0.7,"pith_summary":"NAS-DETR sets out to show that the hardest part of building a sonar detector — choosing a backbone for images that are low-resolution, noisy, and feature-sparse — can be automated by a search that never trains a single candidate network. The search maximizes a differential-entropy proxy for the information capacity of a CNN-transformer backbone, using an evolutionary algorithm under a FLOPs budget, and the winning architecture is trained exactly once inside a full detector. On the URPC2021 and URPC2022 underwater benchmarks this detector reaches 0.538 and 0.492 mean average precision, edging out the RT-DETR baseline by 2.5% and 3.4% at 71.2–73.8 frames per second. If the training-free proxy really ranks architectures the way trained accuracy does, then architecture search for detection in degraded imagery becomes nearly free: the only expensive step left is the single training of the chosen network.","feed_headline":"Training-free entropy search beats RT-DETR on sonar","feed_subtitle":"The search never trains a candidate network, yet the found backbone tops RT-DETR by 2.5–3.4 percent.","key_machinery":"The load-bearing object is the zero-shot fitness function $Z(G)$: a weighted average of differential-entropy estimates across the six backbone stages, each stage scored as $\\log \\operatorname{var}(h_D) + \\log C_{\\mathrm{in}}$. The variances are propagated analytically — for CNN layers, output variance is the product of kernel-area and channel factors across layers, and for transformer blocks the derivation treats the softmax attention map as a near-linear scaling matrix, yielding $\\sigma^2(y) \\approx d_{\\mathrm{model}}^2\\, d_{\\mathrm{feedforward}}\\, S^2\\, \\sigma^2(x)$ — so the differential-entropy upper bound $\\frac{1}{2}\\ln(2\\pi e\\sigma^2)$ can be maximized without any training. A numerical scale-normalization step $F' = F/\\sigma(F)$ keeps the variance finite in deep stacks, with a correction factor relating scaled and unscaled variance. An evolutionary algorithm then mutates kernel width, depth, channels, bottleneck ratio, hidden dimension, and feedforward dimension over 20,000 rounds, keeping the top-ranked individuals under a FLOPs ceiling, and the single highest-scoring backbone is trained once, inside the full detector.","core_discovery":"The paper's central claim is that a zero-shot neural architecture search grounded in the maximum-entropy principle — maximizing the differential entropy of output feature maps, estimated by propagating the variance of Gaussian-initialized activations through CNN and transformer blocks — finds a CNN-transformer hybrid backbone better matched to sonar imagery than backbones designed for optical images. Wrapped in a Deformable-DETR-style decoder with a feature pyramid, query selection, a denoising training task, and a hybrid Varifocal-plus-GIoU loss, the NAS-found backbone achieves what the authors report as state-of-the-art results on URPC2021 (0.538 mmAP) and URPC2022 (0.492 mmAP), surpassing RT-DETR by 2.5% and 3.4% while running at 71.2–73.8 FPS, or 288–294.9 FPS after TensorRT quantization. The paper further claims that the proxy is interpretable: Spearman rank correlation shows depth strongly drives the score ($\\rho \\approx 0.83$–$0.90$), width and kernel size contribute positively, and transformer dimensions correlate with the score only when CNN capacity is held fixed, which the authors attribute to FLOPs-induced competition between the two module types.","pith_inferences":["Because the fitness score is never compared with trained accuracy in the paper, the natural next experiment is to train a sample of searched backbones spanning the full range of $Z(G)$ and check that their measured mAP ranking matches the entropy ranking; that would turn the proxy from an assumption into a validated design tool.","The variance-propagation formula makes entropy grow multiplicatively with depth, which suggests the search is effectively exploring a depth-versus-width Pareto frontier under the FLOPs constraint; the same optimization could be recast with an explicit Pareto objective.","The recipe — entropy proxy, evolutionary search, one final training run — is domain-agnostic in principle, so it should transfer to other low-resolution, high-noise detection tasks such as medical ultrasound, radar, or degraded optical imagery; rerunning the identical search on one such dataset would test that directly.","The best entropy-weight setting differs between the two datasets (A1 on URPC2021, A2 on URPC2022), implying the six stage weights act as dataset-dependent hyperparameters; treating them as part of the search space, rather than fixed choices, would make the method fully automatic."],"forward_implications":["The detector reports state-of-the-art mean average precision on both benchmarks (0.538 and 0.492 mmAP), with the winner on each dataset coming from a different entropy-weight setting (A1 for URPC2021, A2 for URPC2022).","Stepwise ablations (RT-DETR → MAE-DETR → NAS-DETR) improve mmAP at each step, which the paper reads as evidence that the entropy-driven search adds real gain on top of the detector design.","The searched backbone keeps the full detector real-time at 71.2–73.8 FPS (288–294.9 FPS after TensorRT quantization), so the accuracy gain does not sacrifice deployability.","Spearman analysis indicates that backbone depth is the dominant driver of the search score ($\\rho = 0.826$–$0.904$), so the search objective implicitly prioritizes depth within the FLOPs budget."],"supporting_citations":[{"why":"Supplies the multi-scale deformable-attention decoder that NAS-DETR's decoder closely follows.","marker":"[30]"},{"why":"Source of the maximum-entropy zero-shot NAS approach for detection, and the MAE-DETR baseline used in the ablations.","marker":"[41]"},{"why":"Provides the purely mathematical entropy-maximization framework for network design that motivates the fitness function.","marker":"[42]"},{"why":"Cited as the source of the graph representation of a network and of the claim that the differential-entropy upper bound constrains representation capacity.","marker":"[43]"},{"why":"The real-time detector baseline (RT-DETR) whose backbone this work replaces and whose accuracy it surpasses on both datasets.","marker":"[46]"},{"why":"Source of the query-denoising training idea that becomes the additional denoising loss term.","marker":"[49]"},{"why":"Provides the URPC2021 sonar benchmark and the official train/test split used for evaluation.","marker":"[50]"},{"why":"Provides the URPC2022 sonar benchmark with its ten-class extension (including ROV and plane).","marker":"[51]"},{"why":"Supplies the Spearman rank-correlation test used to analyze which architecture parameters drive the entropy score.","marker":"[52]"}],"fun_headline_variants":["Zero-shot entropy NAS finds sonar-optimal DETR","Entropy-guided NAS tops RT-DETR on sonar","Sonar detection: NAS without training beats RT-DETR","Max-entropy search yields better sonar DETR","Zero-shot NAS outdoes RT-DETR in sonar"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search rests on the assumption that the training-free entropy score ranks candidate backbones in the same order as their accuracy after real training, and the paper never checks that ranking against even one trained network.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot entropy NAS finds sonar-optimal DETR","Entropy-guided NAS tops RT-DETR on sonar","Sonar detection: NAS without training beats RT-DETR","Max-entropy search yields better sonar DETR","Zero-shot NAS outdoes RT-DETR in sonar"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2883,"prompt_tokens":1070,"completion_tokens":1813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":1728}},"tokens_in":686,"tokens_out":1813,"duration_ms":11314,"temperature":1.0,"reasoning_tokens":1728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:37:01.066052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ten to twenty backbone architectures sampled across the full range of fitness scores $Z(G)$ inside the otherwise identical detector, and compute the rank correlation between $Z(G)$ and the resulting mean average precision; if the correlation is weak or negative, the entropy proxy is miscalibrated and the reported gains come from the detector components or the FLOPs prior rather than from the search. A second, targeted check would measure the actual variance of the transformer block output at initialization and compare it with the paper's formula $\\sigma^2(y) \\approx d_{\\mathrm{model}}^2\\, d_{\\mathrm{feedforward}}\\, S^2\\, \\sigma^2(x)$; a large mismatch would confirm that treating softmax attention as a linear scaling matrix breaks the derivation.","supporting_citations":[{"cited_title":"Deepmad: Mathematical architecture design for deep convolutional neural network","cited_arxiv_id":null,"evidence_quote":"Provides the purely mathematical entropy-maximization framework for network design that motivates the fitness function."},{"cited_title":"Detrs beat yolos on real-time object detection","cited_arxiv_id":null,"evidence_quote":"The real-time detector baseline (RT-DETR) whose backbone this work replaces and whose accuracy it surpasses on both datasets."},{"cited_title":"Dn-detr: Accelerate detr training by introducing query de- noising","cited_arxiv_id":null,"evidence_quote":"Source of the query-denoising training idea that becomes the additional denoising loss term."},{"cited_title":"Dual spatial attention network for underwater object detec- tion with sonar imagery.IEEE Sensors Journal, 24(5):6998–7008,","cited_arxiv_id":null,"evidence_quote":"Provides the URPC2021 sonar benchmark and the official train/test split used for evaluation."},{"cited_title":"A dataset with multibeam forward-looking sonar for underwater object detection.Scientific Data, 9(1):739, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the URPC2022 sonar benchmark with its ten-class extension (including ROV and plane)."},{"cited_title":"The proof and measurement of associa- tion between two things.The American journal of psychology, 100(3/4):441–471, 1987","cited_arxiv_id":null,"evidence_quote":"Supplies the Spearman rank-correlation test used to analyze which architecture parameters drive the entropy score."}],"review_version":1}