{"id":"c0152aed-889e-4ecf-a57d-5ab26ad71994","arxiv_id":"2506.22517","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A comparative benchmark of YOLOv11, YOLOv12 and RF-DETR for container damage detection shows mixed results, with RF-DETR favored only on three hand-picked hard examples.","lead":"This paper compares three computer vision models for spotting damage on shipping containers using 278 annotated images. It finds that the YOLO models score higher on a standard accuracy measure, while the RF-DETR model seems better on a few unusually damaged containers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RF-DETR superiority claim rests on three unlabeled test images; detection counts without ground truth cannot support an accuracy claim, and it contradicts the paper's own mAP/recall table.","rationale":"The reader's verdict is REJECT, and I agree that the paper should not be accepted as supporting RF-DETR superiority. The reader's weakest assumption focuses on representativeness and chance: three hand-picked images cannot establish generalizable out-of-distribution performance. My strongest concern is more basic: even for these three images, the comparison is not a valid accuracy comparison because no ground-truth boxes are reported for them. Detection counts at a confidence threshold measure how often a model emits boxes, not whether those boxes correspond to real damage. A model with lower mAP/recall on the held-out set could still produce more high-confidence boxes on selected images; without ground truth, those boxes cannot be credited as correct detections. Thus the load-bearing condition for the headline claim is missing twice: the qualitative evaluations are unvalidated, and the sample is too small and non-random. This does not attack the authors' integrity; it is a straightforward measurement gap. The reader and I partially agree: we both identify the three hand-picked images as the weak foundation, but I locate the primary failure in the absence of ground truth rather than in chance or selection alone. The verdict should remain REJECT (UNCHANGED) because the central claim is not supported by the evidence presented.","tokens_in":5465,"tokens_out":2857,"duration_ms":33461,"concrete_test":"Have domain experts annotate the ground-truth damaged sections in the three test images (Figures 10–12). Run all three models at the reported 50% confidence threshold and compute per-image precision, recall, and AP against those annotations instead of raw detection counts. Then sample additional rare-damage images under a pre-registered protocol (e.g., all images in a held-out rare-damage pool) and repeat the comparison. If RF-DETR's extra detections are mostly false positives, or if the ranking flips on a larger sampled set, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that RF-DETR outperforms YOLOv11 and YOLOv12 on 'not-so-common' damaged containers — rests on Tests 1–3 in Section 5. In those tests, the paper reports only the number of predicted damaged sections and their confidence values (e.g., RF-DETR detecting 3 sections at 54%, 55%, and 81% in Test 3). It never reports ground-truth annotations for these three images, so there is no way to know whether the extra RF-DETR boxes are true detections or false positives. On out-of-distribution images, high-confidence boxes can easily be spurious; counting boxes at a 50% confidence threshold therefore measures a model's tendency to fire, not its detection accuracy. The claim is also in tension with the quantitative results: RF-DETR has the lowest mAP@50 (77.7%) and lowest recall (73%) on the 19-image test set, so the three hand-picked images are doing all the work for the headline conclusion. For the claim to hold, the observed qualitative differences would need to reflect genuine, ground-truth-verified detection ability, not selection of favorable images or detectors that simply produce more boxes. That condition is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a comparative benchmark of three recent object detectors—YOLOv11, YOLOv12, and RF-DETR—for shipping container damage detection. The authors train all three models on a dataset of 278 annotated images (234/25/19 train/validation/test split) and report in Table 1 that YOLOv11 and YOLOv12 achieve 81.9% mAP@50, while RF-DETR achieves 77.7%, with precision and recall also tabulated. The paper then tests the models on three additional images (a fire-damaged container, two stacked fire-damaged containers, and a bent container) and, based on the number of predicted damaged sections and their confidence scores, concludes that RF-DETR outperforms the other two models for 'not-so-common' damage types. This conclusion is in tension with the quantitative table and is not supported by the qualitative evidence as presented.","tokens_in":5739,"tokens_out":5187,"duration_ms":58134,"significance":"If the results were fully supported, this comparison would be a useful practical reference for selecting a damage-detection model in container logistics, and the choice of three recent architectures (YOLOv11, YOLOv12, RF-DETR) is timely. The application domain is genuinely important, and the paper makes a concrete attempt to provide both quantitative and qualitative comparisons. However, the study is not reproducible as written: no dataset link, no training hyperparameters, no inference details, no error bars, and no ground-truth annotations for the qualitative tests are provided. The central 'RF-DETR superiority' claim rests on a three-image visual comparison that directly contradicts the reported mAP. As it stands, the work does not meet the evidentiary standard for a comparative study.","major_comments":[{"comment":"The abstract and Section 6 claim that RF-DETR 'outperformed the others overall' and exhibited 'superiority' for not-so-common damaged containers, but this is directly contradicted by Table 1, where RF-DETR has the lowest mAP@50 (77.7%) and the lowest recall (73%), while YOLOv11 and YOLOv12 both reach 81.9% mAP@50. Section 5 itself states that 'there is no clear winner,' which is inconsistent with the abstract and conclusion. The headline claim is therefore not supported by the paper's own quantitative evidence.","section":"§6 Conclusion and Abstract vs. Table 1"},{"comment":"The three-image evaluation is not a valid accuracy comparison. No ground-truth annotations are provided for the fire-damaged, stacked, or bent container images; the paper only reports the number of predicted sections and confidence values (e.g., Test 3: RF-DETR detects 3 sections at 54%, 55%, and 81%). Without ground truth, counting detections above a 50% confidence threshold cannot establish detection accuracy—the extra boxes may be false positives. Additionally, no sampling protocol or selection criterion is given for these three images, so the qualitative conclusion cannot be generalized to 'not-so-common' container damage.","section":"§5, Tests 1–3"},{"comment":"The study is missing essential reproducibility and statistical information. There is no dataset identifier or link, no description of image sources or annotation protocol, no training hyperparameters (epochs, batch size, input resolution, optimizer, learning rate), no inference hardware, and no confidence intervals or repeated trials. With only 19 test images, the difference between 81.9% and 77.7% mAP@50 may be within run-to-run noise, and the paper does not report any measure of variance or significance. Without this information, neither the quantitative comparison nor the qualitative claim can be independently verified.","section":"§3 and §5, Table 1"}],"minor_comments":[{"comment":"There are numerous typographical and grammatical errors, for example 'overtime' in the abstract should be 'over time,' 'prolonged expose' should be 'prolonged exposure,' 'fully bend container' should be 'fully bent container,' and 'displayed showed strong performance' is redundant.","section":"Throughout"},{"comment":"The text describes RF-DETR as a 'single-shot object detector' based on CNN in the introduction to Section 2, but RF-DETR is a transformer-based detector built on DINOv2 and Deformable DETR; this is not a CNN single-shot model. Please clarify the architectural distinction.","section":"§2.3 and §2 introduction"},{"comment":"The mAP@0.5 formula is malformed: the summation index is not defined, the notation 𝛰𝜋(AP𝑢) is not standard, and the equation rendering is broken. A correct definition of average precision and the averaging procedure should be provided.","section":"§4.4"},{"comment":"Figures 10–12 are referenced as showing the qualitative test results, but the images appear not to be included in the submitted text, so the reader cannot visually inspect the detections that support the qualitative claims.","section":"Figures 10–12"},{"comment":"Table 1 has no caption and does not state whether the reported mAP, precision, and recall values are from the best epoch, the last epoch, or a selected checkpoint; this should be specified.","section":"Table 1"},{"comment":"Several references are incomplete or informal, such as [2] (arXiv-only, no version), [4] (no publication venue details), and [5] (GitHub URL without a formal citation); please format the bibliography consistently.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an unreviewed technical report. The main practical recommendation—that RF-DETR is best for uncommon damage—contradicts the paper's own test-set numbers, and the supporting evidence is anecdotal. While the topic is relevant to the journal, the evaluation must be substantially strengthened (ground-truth annotations, more test images, statistical error bars, and reproducibility details) before the claims can be considered credible. I would not support publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on the YOLOv11/YOLOv12/RF-DETR container damage paper.\n\nWhat's actually new: a direct three-way comparison of these detectors on a 278-image container damage dataset, with a clean mAP/precision/recall table. That's a modest but real data point. The paper also gives decent summaries of the three architectures and is honest about the mixed quantitative outcome.\n\nThe weak spot is the qualitative claim. The abstract and conclusion say RF-DETR was superior on 'not-so-common' damaged containers, but that claim rests on three hand-picked test images for which no ground truth is provided. Reporting that RF-DETR fired more boxes at higher confidence says nothing about accuracy—those extra boxes could be false positives. And it's in direct tension with the table, where RF-DETR has the lowest mAP@50 (77.7%) and recall (73%). The three images are doing all the work for the headline, and they aren't labeled.\n\nOther soft spots: the dataset isn't linked, there are no error bars, and training details are thin. The sample size is small (19 test images). These are minor-to-moderate, but they compound the central problem.\n\nNet: the quantitative benchmark is a reasonable starting point, and the paper is worth a skim for the numbers. But the support for the RF-DETR superiority claim is not there. A revision that adds ground-truth annotations for the qualitative tests, more images, error bars, and a dataset link could make this a solid technical report. As is, I'd treat it as a preliminary comparison, not a definitive one.\n\nFor peer review: it's not a desk-reject on substance—the question is legitimate—but it needs a serious referee to ask for the missing evidence. I'd recommend sending it to a small applied venue with the expectation of major revisions, or inviting a resubmission with the fixes.","headline":"A modest three-model benchmark on container damage that is honest about its quantitative results but overclaims RF-DETR superiority from three unlabeled test images.","tokens_in":6175,"tokens_out":4153,"would_cite":false,"duration_ms":37469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper compares YOLOv11, YOLOv12, and RF-DETR for detecting damaged shipping containers and finds that RF-DETR identifies unusual damage more reliably despite scoring lower on the standard mAP@50 benchmark.","keywords":["container damage detection","object detection","YOLOv11","YOLOv12","RF-DETR","mAP","precision","recall"],"falsifier":"Run all three models, at identical confidence thresholds, on a pre-specified set of 50 or more rare damage container images (fire, bent, crushed) not used for training, and compute per-class recall and detection confidence; if RF-DETR does not consistently detect more damaged sections than YOLOv11/12, the central claim is refuted.","tokens_in":5309,"feed_emoji":"📦","tokens_out":6505,"duration_ms":59259,"temperature":0.7,"pith_summary":"This paper compares three recent object detectors—YOLOv11, YOLOv12, and RF-DETR—on the task of finding damaged shipping containers. Using 278 annotated container images, the author trains all three models and finds a mixed result: YOLOv11 and YOLOv12 reach an mAP@50 of 81.9% while RF-DETR trails at 77.7%. When the models are then tested on three unusual damage cases—a fire-damaged container, two stacked fire-damaged containers, and a bent container—RF-DETR detects more damaged sections with higher confidence than the other two. The paper argues that RF-DETR is therefore better suited for spotting not-so-common container damage, even though it loses on the standard benchmark metric.","feed_headline":"RF-DETR finds rare container damage that YOLO models miss","feed_subtitle":"Three tough fire and bend images show higher-confidence hits, even though YOLO scores higher on mAP@50.","key_machinery":"The comparison is carried by three model families and a fixed evaluation protocol. YOLOv11 and YOLOv12 are CNN-based single-shot detectors: YOLOv11 uses C3k2 blocks in the neck and head, while YOLOv12 introduces area attention and residual efficient layer aggregation networks (R-ELAN). RF-DETR replaces the anchor-based head with a transformer encoder-decoder that directly predicts boxes and classes, built on a vision-transformer backbone. The protocol trains all models on the same 234-image training set, validates on 25 images, and evaluates on 19 test images using mAP@50, precision, and recall, followed by three hand-picked out-of-distribution images meant to represent rare damage.","core_discovery":"The paper's central claim is that RF-DETR, a transformer-based detector, outperforms YOLOv11 and YOLOv12 when the input is a not-so-common damage type. On three out-of-dataset test images—a fully fire-damaged container, two stacked fire-damaged containers, and a fully bent container—RF-DETR localized more damaged sections with confidence above 90% in the first case, 93% in the second, and three sections at 54%, 55%, and 81% in the third. YOLOv11 and YOLOv12 each missed sections or reported lower confidence on the same images. The author interprets this as RF-DETR having stronger generalization to unusual damage, despite its lower mAP@50 score on the 19-image test set.","pith_inferences":["Three images are far too few to establish a reliable ranking for rare damage; a reader should treat the RF-DETR advantage as a promising signal, not a settled result.","The qualitative tests compare detection confidence and section counts, not localization accuracy against ground truth; RF-DETR's high confidence could coexist with worse bounding-box precision on a larger sample.","A natural follow-up is to use RF-DETR to pseudo-label rare-damage images and fine-tune YOLOv11/12 on them, potentially combining strong mAP with better coverage of unusual damage.","Before deployment, the models should be tested on a pre-registered set of rare damage images with repeated runs and statistical comparison, since the current protocol cannot distinguish true capability from chance."],"forward_implications":["If RF-DETR's generalization advantage is real, safety-critical container inspection should prefer transformer-based detectors over CNN-based ones for rare damage types, even when standard mAP is lower.","Standard mAP@50 alone would have picked YOLOv11 as the winner; the qualitative tests show that benchmark ranking can mislead deployment choices for out-of-distribution damage.","The mixed result implies that no single model dominates: YOLOv11 has the highest precision (91.9%), YOLOv12 the highest recall (78.4%), and RF-DETR the highest confidence on unusual cases.","The paper's answer to RQ2 is yes: all three deep learning models are strong enough to be practical alternatives to manual container inspection."],"supporting_citations":[{"why":"Supplies the YOLOv11 architecture overview, including the C3k2 and C2PSA blocks the comparison relies on.","marker":"[1]"},{"why":"Provides YOLOv11 benchmark figures and documentation used as the baseline for that model.","marker":"[3]"},{"why":"Defines the attention-centric YOLOv12 architecture with area attention and R-ELAN.","marker":"[5]"},{"why":"Reports RF-DETR versus YOLOv12 performance and is the source for RF-DETR's design and mAP claim.","marker":"[8]"},{"why":"Introduces the DETR transformer detection framework that RF-DETR builds on.","marker":"[10]"},{"why":"Prior container damage detection with YOLO-NAS, establishing the task baseline and expected mAP.","marker":"[11]"}],"fun_headline_variants":["RF-DETR shines on rare container damage despite lower mAP","For uncommon damage RF-DETR outperforms YOLOv11 and YOLOv12","YOLO wins mAP, but RF-DETR wins on rare damage cases","RF-DETR better at detecting unusual container damage than YOLO","RF-DETR spots rare container damage YOLO misses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that RF-DETR is better on not-so-common damage rests on the three hand-picked images being representative of such damage and the detections reflecting true capability rather than chance.","fun_headline_variants_meta":{"raw":{"variants":["RF-DETR shines on rare container damage despite lower mAP","For uncommon damage RF-DETR outperforms YOLOv11 and YOLOv12","YOLO wins mAP, but RF-DETR wins on rare damage cases","RF-DETR better at detecting unusual container damage than YOLO","RF-DETR spots rare container damage YOLO misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3271,"prompt_tokens":962,"completion_tokens":2309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2209}},"tokens_in":578,"tokens_out":2309,"duration_ms":16634,"temperature":1.0,"reasoning_tokens":2209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:26:20.405013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all three models, at identical confidence thresholds, on a pre-specified set of 50 or more rare damage container images (fire, bent, crushed) not used for training, and compute per-class recall and detection confidence; if RF-DETR does not consistently detect more damaged sections than YOLOv11/12, the central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides YOLOv11 benchmark figures and documentation used as the baseline for that model."},{"cited_title":"Two major improvement factors for Yolov12 – Area Attention and Residual Efficient Layer Aggregation Networks (R-ELAN)","cited_arxiv_id":null,"evidence_quote":"Defines the attention-centric YOLOv12 architecture with area attention and R-ELAN."},{"cited_title":"Area attention,","cited_arxiv_id":null,"evidence_quote":"Reports RF-DETR versus YOLOv12 performance and is the source for RF-DETR's design and mAP claim."},{"cited_title":"Comparative Analysis of Deep Learning Models for Honeybee and Threat Detection at Hive Entrances,","cited_arxiv_id":null,"evidence_quote":"Introduces the DETR transformer detection framework that RF-DETR builds on."},{"cited_title":"End-to-end object detection with transformers,","cited_arxiv_id":null,"evidence_quote":"Prior container damage detection with YOLO-NAS, establishing the task baseline and expected mAP."}],"review_version":1}