{"id":"2e2ce8c2-f58c-484f-8389-9c5f3ab36786","arxiv_id":"2507.11267","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A modified YOLOv5s detector with an added small-object head, BiFPN neck, and custom augmentation reports 99.6% mAP on the DSIAC MWIR military-vehicle dataset for correlated-range testing.","lead":"Researchers modified the YOLOv5 object detector with an extra small-object head, a different feature-fusion neck, and custom image augmentation to detect military vehicles in thermal infrared imagery. On a public US Army dataset of ground vehicles at ranges up to 5 km, the system reports up to 99.6% detection accuracy under matched-range testing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 99.6% and 37.7% mAP numbers rest on an evaluation equation that defines precision incorrectly, and the published target-wise T2 mAP values do not average to the reported 'All' value; without released evaluation code and predictions, the headline result is unsupported.","rationale":"The paper's contribution is an applied engineering result whose headline is an empirical performance number: 99.6% mAP@0.5 on correlated DSIAC MWIR data and 37.7% mAP on decorrelated 3 km data, with an 11.4-point improvement over YOLOv5s. For such a claim to hold, the evaluation metric must be implemented correctly. The paper itself supplies the decisive evidence against this: Equation (1) defines precision as accuracy, and the per-class T2 mAP values in Table 4 do not average to the reported 'All' value under the paper's own Equation (3). Both are internal inconsistencies, not merely disagreements with external consensus. The Table 5 decorrelated value of 73.3% further suggests the reported numbers were not assembled from a single consistent evaluation run. The absence of any released artifacts means an independent check is impossible, so the result must be treated as unverified. The reader's weakest-assumption analysis correctly identified the metric-definition error; the arithmetic inconsistency reinforces it. In good faith, this is not an accusation of misconduct but a statement that the empirical central claim is not currently supported by the manuscript's own evidence. The recommended verdict stays REJECT/UNCHANGED because the central claim fails verification as presented; the paper would need corrected evaluation code, consistent tables, and preferably released predictions to be reconsidered.","tokens_in":8846,"tokens_out":2663,"duration_ms":32316,"concrete_test":"Require the authors to release the exact inference predictions (JSON files) and the evaluation script for the DS1 T1 and T2 protocols. An independent evaluator should recompute per-class AP using standard precision TP/(TP+FP) at IoU=0.5 and mAP as the mean per-class AP per Equation (3), then compare against Table 4. In particular, recompute the Pickup class T2 AP (reported 0.163) and the 'All' row T2 mAP (reported 0.377); if the 'All' value differs from the mean of the four per-class values by more than 1 percentage point after correction, the headline results are invalid as reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical, so the correctness of the evaluation is load-bearing. Equation (1) in Section 4.1 defines Precision as (TP+TN)/(TP+TN+FP+FN), which is accuracy, not precision; the stated simplification to 'TP/All Detections' would only be valid if TN=0, which is not the case in object detection, where background is not scored as a class. If the evaluation code follows the printed equation, every reported precision and mAP value in Tables 4 and 5 is invalid. Independently, the target-wise T2 mAP values in Table 4 (right) are 0.622, 0.214, 0.393, and 0.163; their mean is 0.348, yet the 'All' row reports 0.377, contradicting the definition of mAP in Equation (3) as the mean over classes. The decorrelated rows of Table 5 additionally show a value of 73.3%, which is inconsistent with the T2 mAP of 37.7% in Table 4. No predictions, evaluation script, or trained weights are released, so the discrepancy cannot be resolved by inspection. The 99.6% SOTA claim and the 11.4% improvement claim are therefore unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes YOLOatr, a modified YOLOv5s detector with an added P2 small-object head, a BiFPN neck, and a custom data-augmentation profile, and evaluates it on the public DSIAC MWIR ATR dataset. The authors report state-of-the-art results: 99.6% mAP@0.5 on correlated testing (T1) and 37.7% mAP on decorrelated testing at 3 km (T2), with gains of 0.02 and 11.4 percentage points over baseline YOLOv5s. The central claim is that these architectural and augmentation choices deliver a near-perfect, real-time-capable ATR system that generalizes to unseen ranges.","tokens_in":9169,"tokens_out":2775,"duration_ms":30296,"significance":"If the results were reliable, the paper would provide a practically useful engineering contribution: it uses a public military-relevant benchmark, reports a lean architecture with fast inference, and compares correlated and decorrelated testing. However, the evaluation has load-bearing methodological flaws. Equation (1) defines precision as accuracy, the target-wise T2 mAP scores do not average to the reported 'All' value, Table 5 contains a 73.3% decorrelated mAP that contradicts Table 4, and no code or predictions are released. These issues invalidate the quantitative claims as written. The paper's significance is therefore conditional on a complete re-evaluation and data release; in its current form it cannot substantiate the claimed SOTA performance.","major_comments":[{"comment":"Equation (1) defines Precision as (TP+TN)/(TP+TN+FP+FN), which is accuracy, not precision. The stated simplification to TP/All Detections is only valid when TN=0, which does not hold in object detection where background is not a scored class. If the evaluation code follows this equation, every precision and mAP value in Tables 4 and 5 is invalid, including the headline 99.6% and 37.7% figures. The central claim therefore rests on an unsupported quantitative basis.","section":"§4.1, Eq. (1)"},{"comment":"The target-wise T2 mAP values are 0.622 (T72), 0.214 (BTR70), 0.393 (SUV), and 0.163 (Pickup). Their mean is 0.348, not the 0.377 reported in the 'All' row. This directly contradicts the definition of mAP in Eq. (3) as the mean over classes. The discrepancy suggests the numbers were computed or assembled inconsistently, and it cannot be resolved from the paper alone.","section":"§4.3, Table 4 (Right)"},{"comment":"Table 5 lists decorrelated testing results for other methods with mAP values of 73.3% (e.g., Cuellar et al., YOLOv3 (SF), testing at 5.0 km after training at 4.0–4.5 km). If those numbers are comparable to the paper's T2 protocol, they are far higher than the reported 37.7% mAP for YOLOatr in Table 4, contradicting the claim that YOLOatr achieves SOTA decorrelated performance. If they are not comparable, the table lacks the necessary protocol annotations to make any comparison meaningful. Either way, the comparative claims in Section 5 are internally inconsistent.","section":"§5, Table 5"},{"comment":"All experiments appear to be single runs on a single train/test split, with no error bars, repeated runs, or statistical tests. The claimed improvement over YOLOv5s on correlated T1 testing is only 0.002 in mAP (0.996 vs 0.994), a difference that is well within the random variation expected for a single split. Without variance estimates, the 'near-perfect' and 'SOTA' claims are not supported. This issue is load-bearing because the small gain is presented as the main correlated-range result.","section":"§4.3, Tables 1 and 2"},{"comment":"No code, evaluation script, ground-truth predictions, or trained weights are released. Given that Eq. (1) appears to document an incorrect metric and that the reported numbers are internally inconsistent, the absence of release prevents any external verification of the results. Reproducibility is a necessary condition for the empirical claims to be accepted.","section":"General reproducibility"}],"minor_comments":[{"comment":"The phrase 'two learning methodologies two learning methodologies' duplicates words; it should be corrected to 'two learning methodologies.'","section":"§3.2"},{"comment":"Table 1 states '13 different MWIR Visible,' but the text and the target list describe 10 vehicles and 2 humans, which totals 12 classes. The discrepancy between the table and the text should be resolved.","section":"§3.1, Table 1"},{"comment":"The reference '[Millikan et al., 2018]' appears twice with different content: once as a URL to the DSIAC database and once as a CVPR workshop paper. The database URL should be cited separately or removed.","section":"References"},{"comment":"The row 'Performance Gain +0.02% +11.4%' should use percentage points rather than percent when describing differences between mAP values (e.g., '0.2 percentage point' or '11.4 percentage points').","section":"§4.3, Table 4"},{"comment":"The caption contains a duplicated phrase: 'Target detection at 5000m (Bottom Right, Detection at 5000m range (Bottom Right)'. This should be cleaned up.","section":"§Figure 5 caption"},{"comment":"The BiFPN neck is mentioned but the implementation details (e.g., whether learnable weights or a specific number of repeats are used) are not provided, making the architectural modification difficult to reproduce.","section":"§3.2"}],"recommendation":"reject","confidential_remarks":"The technical issues in the paper are severe enough that the headline claims cannot be accepted in their current form. If the authors were to correct the evaluation metric, re-run the experiments with proper per-class AP aggregation, reconcile the contradictory tables, and release code and predictions, a resubmission might be considered. However, as submitted, the manuscript's core quantitative contribution is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this one for the wrong reason: it is a clean example of how evaluation sloppiness can sink an otherwise plausible engineering report. The authors take YOLOv5s, add a P2 head and a BiFPN neck, tune an augmentation profile, and test on the DSIAC MWIR dataset with both correlated and decorrelated splits. Those modifications are well-established in the object-detection literature; the only new thing is applying them to this ATR benchmark. The paper is clearly written and the limitations section is honest about what was not tried. The dataset choice is sensible and the correlated/decorrelated distinction is the right way to think about generalization in this domain.\n\nThe problem is that the numbers do not hold together. Equation (1) defines precision as (TP+TN)/(TP+TN+FP+FN), which is accuracy, not precision; the simplification to TP/All Detections contradicts that expression. If the evaluation code followed the printed formula, every precision and mAP in the paper is suspect. On top of that, the target-wise T2 mAP values in Table 4 (0.622, 0.214, 0.393, 0.163) average to 0.348, not the reported 0.377. And Table 5 shows a 73.3% figure for decorrelated settings that does not match the 37.7% in Table 4. No code, predictions, or trained weights are released, so these discrepancies cannot be checked by an interested reader. The headline 99.6% mAP is really a 0.2-point gain over an already saturated correlated test, while the meaningful 11.4-point gain on decorrelated data is tied to a number that is internally inconsistent.\n\nSo the central empirical claim is unsupported as written. That is a load-bearing flaw, not a cosmetic one. Still, the paper is not nonsense; the authors engage the literature and identify the right domain challenges. It just is not ready for publication without a corrected evaluation, released artifacts, and a careful reconciliation of the tables. For a reading group, it would make a useful cautionary tale about evaluation pitfalls, but I would not cite it and I would not spend referee time on it as-is. My recommendation: desk reject, but invite a resubmission with a corrected protocol and code release.","headline":"A straightforward YOLOv5s tweak applied to DSIAC, but the evaluation is too inconsistent to support the SOTA claim.","tokens_in":9686,"tokens_out":2174,"would_cite":false,"duration_ms":25178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"YOLOatr, a modified YOLOv5s, reaches 99.6% mAP on correlated thermal images and 37.7% mAP on longer-range test images.","keywords":["Automatic Target Recognition","Thermal Infrared Imagery","YOLOv5","Small Object Detection","BiFPN","Data Augmentation","DSIAC MWIR","Decorrelated Testing"],"falsifier":"Recompute mAP@0.5 from the saved test predictions using the standard definitions (precision = TP/(TP+FP), recall = TP/(TP+FN)) and compare with Table 4; if the evaluation used the paper's printed formula, the numbers cannot stand. Independently, averaging the four target-wise mAP rows in Table 4 does not reproduce the reported 0.377 'All' value, so that arithmetic discrepancy is checkable directly from the table.","tokens_in":8691,"feed_emoji":"🎯","tokens_out":15024,"duration_ms":160434,"temperature":0.7,"pith_summary":"The paper proposes YOLOatr, a detector for automatic target detection and recognition in thermal infrared imagery, built by modifying YOLOv5s with an extra P2 small-object head, a BiFPN feature-fusion neck, and a custom augmentation profile. It claims YOLOatr raises mAP@0.5 on the DSIAC MWIR benchmark from 99.4% (YOLOv5s) to 99.6% on correlated test data, and from 27.1% to 37.7% on the decorrelated 3 km test. The aim is to show that a lean, single-frame, single-stage detector can recognize small, low-resolution thermal targets in clutter at long ranges in real time, a setting where the authors say existing detectors underperform. If the claimed results hold, the practical consequence is that on-board tactical ATR at kilometer ranges is feasible without large computational resources.","feed_headline":"Modified YOLOv5 hits 99.6% on thermal target recognition","feed_subtitle":"A small-object head and weighted feature fusion lift longer-range test scores by 11.4 points.","key_machinery":"The mechanism is a modified YOLOv5s architecture. An extra P2 detection head operates on a higher-resolution feature map so targets that occupy only a few pixels get a dedicated prediction scale; the default PANet neck is replaced with a BiFPN neck that learns weighted bidirectional cross-scale feature fusion. Around this, a custom augmentation profile keeps brightness and contrast variation, keeps mosaic augmentation low so small targets are not shrunk, disables shear, and sets high mixup and copy-paste values. Training is done from scratch on thermal images rather than from ImageNet-pretrained weights, which the paper found slightly better for this modality. The paper interprets this combination as the reason YOLOatr detects small targets in clutter and improves decorrelated-range generalization.","core_discovery":"The central discovery claimed is architectural: adding a small-object head and weighted cross-scale feature fusion to YOLOv5s produces a detector that keeps 99.6% mAP within the training range and generalizes substantially better when the test range is longer. The decisive evidence is the decorrelated protocol, where YOLOv5s collapses to 27.1% mAP while YOLOatr reaches 37.7% mAP, a gain the paper attributes to the extra P2 head and BiFPN neck rather than to more parameters. The gain is uneven across classes, with the T72 tank at 62.2% mAP and the Pickup truck at 16.3% mAP at 3 km, which the paper ties to target size and thermal signature. The paper also reports 110 fps inference with about 7.1 million parameters, making the claimed accuracy compatible with real-time operation.","pith_inferences":["Because the paper leaves false-alarm rate for future work, an operational evaluation should use false-alarm-rate versus probability-of-detection curves rather than mAP alone.","An ablation that removes only the BiFPN neck while keeping the P2 head would test which component carries the decorrelated-range gain; the paper reports the combined model but not this decomposition.","The same head-and-augmentation recipe could be applied to visible-band small-object benchmarks; the paper's evidence is limited to the MWIR dataset, so cross-modality generalization remains untested.","Stricter evaluation at a higher IoU threshold or under lower signal-to-clutter ratios would test whether the near-perfect correlated score reflects precise localization or mainly generous overlap at the 0.5 threshold."],"forward_implications":["At the claimed 110 fps with roughly 7.1 million parameters and 16.4 GFLOPs, the detector's accuracy is achievable in real time on modest GPU hardware.","Training from scratch on thermal data slightly outperformed ImageNet transfer learning, indicating that RGB pretraining carries little advantage for MWIR automatic target recognition.","Because the gain over YOLOv5s is concentrated in the decorrelated protocol, the extra P2 head and BiFPN neck act mainly as range-generalization improvements rather than as within-range accuracy boosts.","Target-wise results show large class disparities at 3 km, so the model's real-world utility depends on target type: heavy tactical vehicles with strong thermal signatures are far better recognized than civilian vehicles."],"supporting_citations":[{"why":"Supplies the base YOLOv5s model that the paper modifies into YOLOatr.","marker":"[Jocher, 2022]"},{"why":"Provides prior single-stage detector ATR results on the DSIAC dataset that YOLOatr compares against in the performance table.","marker":"[Chen et al., 2021]"},{"why":"Establishes the decorrelated protocol of training at lower ranges and testing at higher ranges, which YOLOatr uses for its T2 evaluation.","marker":"[Cuellar & Mahalanobis, 2021]"},{"why":"Presents TCRNet, the decorrelated-range target detection baseline that the paper's comparison table includes.","marker":"[McIntosh et al., 2020]"},{"why":"Gives the earlier Faster R-CNN versus quadratic correlation filter comparison on the DSIAC dataset that motivates the search for better detectors.","marker":"[Mahalanobis & McIntosh, 2019]"},{"why":"Provides a previous DSIAC target recognition result using autoencoder-generated training images, another baseline the paper cites.","marker":"[Arif & Mahalanobis, 2021]"},{"why":"Demonstrates CNN-based infrared target recognition with limited data, the data-shortage context YOLOatr responds to.","marker":"[d'Acremont et al., 2019]"}],"fun_headline_variants":["YOLOatr: thermal ATR hits 99.6% mAP with P2 head","Modified YOLOv5 beats baseline by 10.6 points at long range","Small-object head and BiFPN lift thermal detector to 99.6%","YOLOatr: real-time thermal detection with 99.6% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the evaluation code computed precision and mean average precision correctly, because the paper's printed precision formula is actually accuracy and its two tables disagree on the decorrelated result.","fun_headline_variants_meta":{"raw":{"variants":["YOLOatr: thermal ATR hits 99.6% mAP with P2 head","Modified YOLOv5 beats baseline by 10.6 points at long range","Small-object head and BiFPN lift thermal detector to 99.6%","YOLOatr: real-time thermal detection with 99.6% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000365,"raw_usage":{"total_tokens":1974,"prompt_tokens":964,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":917}},"tokens_in":580,"tokens_out":1010,"duration_ms":10836,"temperature":1.0,"reasoning_tokens":917,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:12:37.389553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute mAP@0.5 from the saved test predictions using the standard definitions (precision = TP/(TP+FP), recall = TP/(TP+FN)) and compare with Table 4; if the evaluation used the paper's printed formula, the numbers cannot stand. Independently, averaging the four target-wise mAP rows in Table 4 does not reproduce the reported 0.377 'All' value, so that arithmetic discrepancy is checkable directly from the table.","supporting_citations":[],"review_version":1}