{"id":"317c154c-d5fc-43d0-9abe-766784eccbbd","arxiv_id":"2501.16469","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark table claims RT-DETR beats YOLOv5, YOLOv8, SSD, and DETR on diabetic retinopathy detection, but the experiment is not reproducible.","lead":"This paper claims RT-DETR, an existing object detection model, outperforms four rivals on diabetic retinopathy lesion detection in the EyePACS dataset. The claim cannot be checked because the paper offers no code, no data, no error bars, and no clear annotation protocol.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states where lesion bounding boxes come from, and EyePACS—as described in Section III-A—has only image-level severity grades; without box labels, Table 1's object-detection metrics cannot be produced or verified.","rationale":"The reader's weakest assumption—that EyePACS contains bounding-box lesion annotations—is exactly the load-bearing issue. The paper's own description in Section III-A refers only to five-level severity grades, so the gap is internal, not just a disagreement with external dataset knowledge. Without a documented source of bounding boxes, the central experimental claim cannot be connected to any ground truth. The other defects (corrupted equations, missing figures, no code, no error bars, and the ablation table's mismatch between optimizer names and learning-rate values) compound the problem but are secondary. Because the dataset-annotation gap invalidates the experimental foundation, I agree with the reader's REJECT verdict and recommend no change.","tokens_in":5846,"tokens_out":2767,"duration_ms":28581,"concrete_test":"Download the exact EyePACS release used (or the Kaggle EyePACS dataset) and inspect its metadata and annotation files for any bounding-box or segmentation labels. If only image-level severity grades are present, Section III-A's claim that the dataset supports object-detection training is false and Table 1 cannot be produced without a separate, undocumented labeling effort. As a second check, request the authors' generated box annotations and reproduce Table 1; if boxes were machine-generated, that pipeline and its validation would need to be released.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To train RT-DETR and compare it against SSD, YOLOv5, YOLOv8, and DETR, each fundus image must have ground-truth bounding boxes for lesions (e.g., microaneurysms, hemorrhages, exudates), with class labels, a defined train/test split, and an evaluation protocol. Section III-A never mentions such annotations: it says EyePACS images carry 'clinical labels' consisting of 5 severity levels (0–4) based on manual diagnosis and that images are 'accurately annotated with the degree of lesions.' Severity grades are image-level labels; they do not specify object locations. The widely distributed EyePACS Kaggle challenge provides only image-level grades, so the necessary boxes are not part of the cited public dataset. The paper also describes the data as 224x224 images with about 35,000 annotated images, which does not match the standard EyePACS release, and no preprocessing or box-generation step is described. Without a documented annotation source, Table 1's metrics (RT-DETR: Precision 0.90, Recall 0.85, mAP50 0.88, mAP50-95 0.76) cannot be connected to any ground truth, and the central claim that RT-DETR outperforms four baselines on diabetic retinopathy detection is unsupported. This is not merely missing reproducibility; it is an internal inconsistency between the dataset description and the object-detection experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes applying the RT-DETR object-detection model to diabetic retinopathy lesion detection using the EyePACS dataset. It compares RT-DETR with YOLOv5, YOLOv8, SSD, and DETR on precision, recall, mAP50, and mAP50-95, reporting that RT-DETR achieves the best scores on all metrics (e.g., Precision 0.90, Recall 0.85, mAP50 0.88, mAP50-95 0.76). The paper also includes a so-called ablation study that varies the learning rate. The method section describes an attention-based NMS-free detector with a combined classification/L1/GIoU loss, but the displayed equations are corrupted and the dataset description does not mention the bounding-box annotations required for object detection.","tokens_in":6090,"tokens_out":4371,"duration_ms":40027,"significance":"If the evaluation were properly grounded, a reproducible comparison of RT-DETR against established detectors on a public medical-image benchmark could be a useful reference for practitioners. The paper has a clear comparative structure and includes metric tables, which are positive features. However, the central finding is not verifiable as written: the dataset is described only in terms of image-level severity labels, no object-level ground truth is described, the equations are unreadable, and the experimental protocol lacks a train/test split or statistical support. The significance of the claimed superiority of RT-DETR is therefore not established.","major_comments":[{"comment":"The dataset description states that EyePACS images are annotated with five severity levels (0-4) based on manual diagnosis, which are image-level clinical labels. The experiment performs object detection, which requires ground-truth bounding boxes and class labels for individual lesions. The paper never states how these boxes were obtained, what lesion classes were used, or how the train/validation/test split was defined. Without such annotations, the metrics in Table 1 (e.g., Precision 0.90, mAP50-95 0.76) cannot be connected to any ground truth, and the central claim that RT-DETR outperforms the four baselines is unsupported.","section":"Section III-A, Table 1"},{"comment":"The three displayed equations are corrupted and unreadable: the attention formula appears as \"V d QKTsoftVKQAttention k )max(),,( ���\", the composite loss appears as \"GIoUGIoULLclscls LLLL ���=��� ��� ���\", and the matching-score formula appears as \"locclsmatch SSS ���=��� ��� ���\". It is impossible to verify the method, the loss weights, or the matching coefficients. This makes the method section unverifiable and blocks reproducibility.","section":"Section II (Method), Eqs. (1)-(3)"},{"comment":"The results are presented as single-run metrics with no error bars, confidence intervals, statistical significance tests, or description of random seeds or repeated trials. There is also no statement of the train/validation/test split or of whether all baselines were trained under identical conditions (e.g., same data splits, pre-processing, and training schedule). The claimed superiority of RT-DETR could therefore be due to random variation or to unequal experimental settings.","section":"Section III-B, Table 1"},{"comment":"The so-called ablation experiment varies only the learning rate. It does not ablate any architectural component of RT-DETR (such as the NMS-free head, the multi-scale feature extraction, or the loss-term weights), so the conclusion that \"an appropriate learning rate\" improves performance does not support the paper's architectural claims about RT-DETR.","section":"Section III-B, Table 2"},{"comment":"The dataset section says the images are \"approximately 35,000 annotated images\" and that \"the size of each image is usually 224x224 pixels, with high resolution and clarity.\" No preprocessing pipeline is described that would produce 224x224 images from the original fundus photographs, and a 224x224 resolution is not evidently \"high resolution.\" This lack of a clear data pipeline compounds the missing-annotation problem.","section":"Section III-A (Datasets)"}],"minor_comments":[{"comment":"Reference [4] is a paper on coyote optimization for diabetic retinopathy grading, not the original RT-DETR paper; reference [9] is Deformable DETR, not the original DETR paper. The manuscript should cite the actual RT-DETR paper (e.g., Zhao et al.) and the original DETR paper (Carion et al.).","section":"References [4] and [9]"},{"comment":"The text refers to \"Figure 1 Model architecture diagram\" and \"Figure 2 Loss function changes with epoch,\" but no actual figures are present in the manuscript.","section":"Figures 1 and 2"},{"comment":"The paper consistently uses the phrase \"target detection\" where the standard term, used in the title and abstract, is \"object detection.\" The terminology should be unified.","section":"Throughout"},{"comment":"The notation \"CWH\" for image size should read C x W x H (or C x H x W), and \"CWHRF\" should be expressed as a tensor in R^{C x W x H}. The current notation is ambiguous.","section":"Section II"},{"comment":"The abstract and conclusion state that RT-DETR achieves \"superior performance\" without acknowledging the absence of error bars or significance tests. The wording should be tempered to match the level of evidence actually reported.","section":"Abstract and Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript has several fundamental issues: the object-detection ground truth is not described, the equations are garbled, the experimental protocol is underspecified, and the citation practice for the core models is inaccurate. Because the central claim depends on unverifiable experiments, I recommend rejection. The paper would need a complete re-evaluation with a properly described annotated dataset, a full experimental protocol, and a corrected related-work section before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the one thing to know: this is a benchmark comparison of RT-DETR against YOLOv5, YOLOv8, SSD, and DETR on EyePACS, and it is not verifiable. The central claim—that RT-DETR wins by 2–4 mAP points—rests on a single table with no error bars, no code, no split description, and no documented ground-truth bounding boxes.\n\nWhat is genuinely useful here is limited. The idea of evaluating RT-DETR on diabetic retinopathy lesion detection is reasonable; RT-DETR is a real model and the comparison set is sensible. The learning-rate sweep in Table 2 at least tries to show a tuning step. That is where the credit ends.\n\nThe soft spots are serious. Section III-A describes EyePACS as an image-level grading dataset with five severity levels, then Section III-B reports object-detection metrics. Those metrics require per-lesion bounding boxes, and the paper never says where they came from. The EyePACS Kaggle release does not include boxes, so either the paper used a modified or privately annotated version of the dataset and did not say so, or the numbers are not connected to any ground truth. Either way, Table 1 is unsupported.\n\nThe method section is also corrupted: the equations in Section II are garbled (the attention formula and loss expression are rendered as unreadable tokens), Figure 1 and Figure 2 are missing, and reference [4] is not the RT-DETR paper—it is an unrelated DR grading paper. That citation error is a serious sign for a paper whose whole subject is RT-DETR. There are also no error bars, no seed repetitions, and no description of train/validation/test partitioning.\n\nIs there a circularity problem? No fitted-parameter derivation, so no circular reasoning in that sense. The problem is simply that the empirical result is not reproducible.\n\nWho is this for? A reader wanting an initial sanity check that RT-DETR can be applied to fundus images might glance at it, but nobody should rely on these numbers. This is not routine-but-weak science; it is an incomplete manuscript. The dataset mismatch alone is enough to reject. If the authors fixed the equations, cited RT-DETR properly, and described an actual box annotation source (or used a dataset that has one), the comparison could be worth a look. As written, it does not deserve referee time.","headline":"Routine RT-DETR benchmark on EyePACS that is not verifiable: no code, no split, no bounding-box annotation source, and the cited dataset description contradicts the reported object-detection metrics.","tokens_in":6679,"tokens_out":2555,"would_cite":false,"duration_ms":23012,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RT-DETR outperforms YOLOv8, YOLOv5, SSD, and DETR on retinal lesion detection metrics.","keywords":["diabetic retinopathy","RT-DETR","object detection","deep learning","EyePACS dataset","Transformer","medical image analysis","lesion detection"],"falsifier":"Open the EyePACS dataset's annotation files: if they contain only image-level severity grades (0-4) and no lesion bounding boxes, then the mAP numbers in the paper cannot be reproduced as reported, and the comparison against YOLOv8, YOLOv5, SSD, and DETR would need an explicit description of how boxes were generated.","tokens_in":5589,"feed_emoji":"👁️","tokens_out":7037,"duration_ms":52245,"temperature":0.7,"pith_summary":"This paper argues that the RT-DETR object detection model, a Transformer-based detector that removes the non-maximum suppression post-processing step, is better suited than strong CNN and Transformer baselines for finding diabetic retinopathy lesions in retinal fundus images. In a five-model comparison on the EyePACS dataset, the authors report RT-DETR as the top performer on precision, recall, mAP50, and mAP50-95, with the largest margins on the stricter mAP50-95 metric. The paper interprets this as evidence that NMS-free end-to-end detection with multi-scale features helps on small, densely packed lesions. A sympathetic reader would take the contribution as a benchmark-style extension: applying RT-DETR to a medical detection task and showing favorable numbers against four established detectors.","feed_headline":"RT-DETR beats YOLO, SSD, DETR on retinal lesion detection","feed_subtitle":"Reported metrics put it ahead of YOLOv8, SSD, and DETR, with the biggest edge on strict IoU thresholds.","key_machinery":"The central object is RT-DETR, a real-time detection Transformer whose detection head replaces the traditional anchor- and NMS-based pipeline with a set of learned object queries and a multi-head self-attention mechanism. The paper describes the attention formulation as $\\text{Attention}(Q,K,V)=\\text{softmax}(\\frac{QK^T}{\\sqrt{d_k}})V$ and a composite loss of cross-entropy classification loss, L1 bounding-box regression loss, and generalized IoU loss, combined with a dynamic matching score that blends classification and localization scores. These components are what the paper credits for enabling the model to localize small, dense lesions without relying on non-maximum suppression.","core_discovery":"The central discovery claimed is that RT-DETR achieves the best detection performance among the compared models on the diabetic retinopathy lesion detection task: precision 0.90, recall 0.85, mAP50 0.88, and mAP50-95 0.76, versus the next-best YOLOv8 at 0.88/0.83/0.86/0.72. The authors attribute this advantage to the model's Transformer-based attention mechanism, its dynamic matching for target assignment, and the lack of an NMS post-processing stage, which they argue is particularly beneficial for small and dense targets. The paper also reports a learning-rate ablation showing that their best configuration uses a learning rate of 0.01, and it concludes that RT-DETR is a promising framework for automated diabetic retinopathy screening and, more broadly, for high-dimensional image analysis.","pith_inferences":["The paper leaves unstated how bounding-box annotations were obtained for EyePACS, whose standard labels are image-level severity grades; if the training used those grades only, the mAP figures depend on an unstated conversion step, so the absolute numbers should be read as indications rather than verified benchmark results.","A direct test would be to rerun the same five-model comparison on a retinal dataset with explicit lesion boxes, or to report a reproducible procedure for deriving pseudo-boxes from the grades.","The claimed NMS-free advantage is not isolated by an ablation; holding the backbone and loss fixed while toggling only NMS would tell whether the benefit comes from the architecture or from the end-to-end matching."],"forward_implications":["If the reported margins hold under proper validation, RT-DETR could serve as a drop-in detection backbone for DR screening pipelines, improving recall on small lesions without a separate NMS stage.","The mAP50-95 gap of 0.04 over YOLOv8 suggests the largest benefit appears under stricter IoU thresholds, which matters for tasks requiring precise boundaries, such as lesion grading.","The NMS-free property could simplify deployment in clinical software, removing a hand-tuned post-processing step from the detection pipeline.","The learning-rate ablation shows the reported advantage depends on tuning; a default or poorly chosen learning rate could erase the gap."],"supporting_citations":[{"why":"Cited for the NMS-free detection property that the paper's argument for RT-DETR's suitability for small, dense lesions relies on.","marker":"[4]"},{"why":"The YOLOv5 baseline that RT-DETR is compared against in the main table.","marker":"[6]"},{"why":"The YOLOv8 baseline, the closest competitor in the reported metrics.","marker":"[7]"},{"why":"The SSD baseline, the weakest performer in the comparison.","marker":"[8]"},{"why":"The DETR-family baseline used to represent Transformer detection in the comparison.","marker":"[9]"}],"fun_headline_variants":["RT-DETR bests YOLOv8, SSD, DETR on retinal lesion detection","RT-DETR outperforms YOLOv8, SSD, DETR on diabetic retinopathy","RT-DETR tops YOLOv8, SSD, DETR on DR lesion detection","RT-DETR surpasses YOLOv8, SSD, DETR on retinal lesions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported results assume that the EyePACS retinal images come with the bounding-box lesion annotations needed to train an object detector and to compute mAP; the paper never states where those boxes came from.","fun_headline_variants_meta":{"raw":{"variants":["RT-DETR bests YOLOv8, SSD, DETR on retinal lesion detection","RT-DETR outperforms YOLOv8, SSD, DETR on diabetic retinopathy","RT-DETR tops YOLOv8, SSD, DETR on DR lesion detection","RT-DETR surpasses YOLOv8, SSD, DETR on retinal lesions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002078,"raw_usage":{"total_tokens":8061,"prompt_tokens":903,"completion_tokens":7158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":7057}},"tokens_in":519,"tokens_out":7158,"duration_ms":38835,"temperature":1.0,"reasoning_tokens":7057,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:02:27.987940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the EyePACS dataset's annotation files: if they contain only image-level severity grades (0-4) and no lesion bounding boxes, then the mAP numbers in the paper cannot be reproduced as reported, and the comparison against YOLOv8, YOLOv5, SSD, and DETR would need an explicit description of how boxes were generated.","supporting_citations":[{"cited_title":"Diabetic retinopathy detection and grading of retinal fundus images using coyote optimization algorithm with deep learning[J]","cited_arxiv_id":null,"evidence_quote":"Cited for the NMS-free detection property that the paper's argument for RT-DETR's suitability for small, dense lesions relies on."},{"cited_title":"A new method based on deep learning to detect lesions in retinal images using YOLOv5[C]//2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)","cited_arxiv_id":null,"evidence_quote":"The YOLOv5 baseline that RT-DETR is compared against in the main table."},{"cited_title":"Diabetic Retinopathy Features Segmentation without Coding Experience with Computer Vision Models YOLOv8 and YOLOv9[J]","cited_arxiv_id":null,"evidence_quote":"The YOLOv8 baseline, the closest competitor in the reported metrics."},{"cited_title":"Ssd: Single shot multibox detector[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I","cited_arxiv_id":null,"evidence_quote":"The SSD baseline, the weakest performer in the comparison."}],"review_version":1}