{"id":"b5092cd1-6ba1-4516-9a8b-a27309779f1a","arxiv_id":"2412.12349","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"On the new ROAD-Almaty dashcam dataset, pretrained RT-DETR reportedly achieves the highest average F1-score of 0.672 at IoU=0.5, ahead of YOLOv8s and YOLO-NAS.","lead":"The paper tests three pretrained object detectors on a new dashcam dataset from Almaty, Kazakhstan, without retraining. It reports that RT-DETR beats two YOLO-family models, with the ranking driven by F1-score at IoU=0.5.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I duplicates Seq1=Seq4 and Seq2=Seq5 for every model and IoU threshold, so the reported averages and ranking double-count the same test data; the central claim's quantitative basis is not established.","rationale":"I read the paper as an empirical benchmark claim: three off-the-shelf detectors are evaluated on a new geographic-domain dataset, and RT-DETR is reported as the best. For that claim to hold, the per-sequence numbers in Table I must be independent, correctly aggregated observations. The duplication of Seq1=Seq4 and Seq2=Seq5 across all models and thresholds is the single most load-bearing weakness because it directly corrupts the averages from which the headline percentages are derived. The reader identified exactly this issue, and I agree. I considered whether another concern—such as the absence of dataset release, lack of error bars, or underspecified test-set construction—might be more fundamental, but those are secondary: even if all other aspects were exemplary, the duplicated rows alone make the reported quantitative comparison unreliable. The paper could be repaired by clarifying whether the duplicated rows are typos or actual sequence reuse; if they are typos, corrected numbers may or may not preserve the ranking. If they reflect reuse, the evaluation protocol itself is invalid. Either way, as submitted, the central claim is not established. I therefore see no reason to change the reader's REJECT verdict; my recommendation is UNCHANGED, with the caveat that the rejection is specifically grounded in this duplication and could be reversed by a corrected, reproducible evaluation.","tokens_in":7519,"tokens_out":1750,"duration_ms":17636,"concrete_test":"Request or reconstruct the per-sequence evaluation logs and the sequence-to-video mapping for ROAD-Almaty. Verify whether Seq4 and Seq5 are physically distinct videos or the same frames under different labels. Then recompute the average F1/IoU at both thresholds using only the unique sequences (e.g., Seq1, Seq2, Seq3, or equivalently remove the duplicated rows) and compare the corrected model ranking against the paper's claims. If the corrected averages yield a different best model, or the reported 46%/27% gaps shrink below the noise level, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that RT-DETR achieves average F1=0.672 at IoU=0.5, outperforming YOLOv8s (0.458) and YOLO-NAS (0.526)—rests entirely on the per-sequence results in Table I. In that table, Seq1 and Seq4 have identical IoU/F1 values for every model at every threshold, and Seq2 and Seq5 are likewise identical. For example, at IoU=0.5, RT-DETR shows 0.890/0.900 for both Seq1 and Seq4, and 0.692/0.517 for both Seq2 and Seq5; the same pattern appears for YOLO-NAS and YOLOv8s. If Seq4 and Seq5 are separate test sequences, it is implausible that all metrics coincide exactly for all models and thresholds; if they are the same sequence listed twice, or if the rows were accidentally duplicated, then the reported averages are computed over only three unique sequences, with two of them double-counted. The resulting ranking and the 46%/27% improvement percentages are therefore not trustworthy as stated. The manuscript does not release the dataset, the sequence definitions, or the evaluation code, so the duplication cannot be resolved from the paper alone. This is a concrete, load-bearing flaw in the central empirical argument, independent of other concerns such as the lack of error bars or the underspecified test-selection procedure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript evaluates the out-of-the-box object detection performance of YOLOv8s, RT-DETR, and YOLO-NAS on a newly collected driving dataset from Almaty, Kazakhstan (ROAD-Almaty). The authors report IoU and F1 scores at thresholds 0.5 and 0.75 for five test sequences and conclude that RT-DETR generalizes best, with an average F1 of 0.672 at IoU=0.5, outperforming YOLOv8s and YOLO-NAS by roughly 46% and 27%. No retraining or domain adaptation is performed; the paper frames the results as evidence about domain generalization in an underrepresented geographic region.","tokens_in":7763,"tokens_out":6494,"duration_ms":56253,"significance":"If the reported results were reliable, the study would provide a useful empirical data point on off-the-shelf detector robustness in Central Asia and a modest extension of domain-generalization evidence to an underrepresented region. The dataset construction and annotation protocol are described in reasonable detail, and the use of standard COCO-style metrics supports comparability. However, the paper makes no methodological contribution, releases no data or code, and its central quantitative claim is compromised by apparent duplicate rows in the main results table. The significance of the contribution is therefore contingent on a corrected and independently verifiable evaluation.","major_comments":[{"comment":"The entries for Seq1 and Seq4 are identical for every model and every IoU threshold (e.g., RT-DETR at IoU=0.5: 0.890/0.900 for both), and the entries for Seq2 and Seq5 are likewise identical (e.g., 0.692/0.517 for RT-DETR at IoU=0.5). If Seq4 and Seq5 are truly distinct sequences, exact equality across all six model-threshold combinations is implausible; if they are the same sequence listed twice, the reported averages and the headline 46%/27% improvement percentages double-count two of the five sequences. The authors must clarify the sequence definitions, correct or remove the duplicate rows, and recompute all averages and percentages.","section":"Sec. IV-A, Table I"},{"comment":"The comparison rests on aggregate F1/IoU values with no measure of uncertainty, no per-sequence frame counts, and no statistical test. With only five sequences (or effectively three unique ones after the duplication issue), the claim that RT-DETR consistently outperforms YOLOv8s and YOLO-NAS is not supported by any variance estimate or significance test. Error bars, per-sequence sample sizes, and a paired comparison (e.g., per-sequence differences) are needed to sustain the ranking.","section":"Sec. IV-B"},{"comment":"The study is framed as an evaluation of domain generalization, but it measures performance only on the target dataset. Without a matched evaluation of the same models on their source-domain data or on a familiar benchmark such as COCO or BDD100K under the same protocol, the observed scores cannot be attributed to domain shift; they are absolute performance levels. A source-domain baseline is required to quantify the generalization gap and to support the conclusion that geographic diversity is the cause of the performance degradation.","section":"Sec. III-B / Sec. V-A"},{"comment":"The manuscript provides no availability statement for the ROAD-Almaty dataset, no sequence definitions, and no evaluation code. Because the paper's entire contribution is empirical and the central table contains an apparent duplication, the results cannot be independently checked or reproduced. The authors should release the per-sequence evaluation results and sequence metadata at minimum, and ideally the dataset and scripts.","section":"Sec. III"}],"minor_comments":[{"comment":"The reference list is inconsistent with the text: [11] is listed as TrafficPredict but cited as ApolloScape in Section II-C, and [20] is cited as a general UDA reference but is a specific residual transfer network paper. Please reconcile all numeric citations with the bibliography.","section":"References [11], [20]"},{"comment":"The description of dividing the dataset into training, validation, and testing sets is confusing because the models are not trained on ROAD-Almaty; clarify that the split is used only for selecting evaluation frames or for future adaptation work.","section":"Sec. III-B1"},{"comment":"Add a Data Availability statement indicating whether the ROAD-Almaty dataset and the evaluation scripts will be made publicly available, and if so, under what license.","section":"Data availability"},{"comment":"There are typographical issues such as 'W AYMO' in Section II-C and 'A Vs' in Section V-C; these should be corrected.","section":"Sec. II-C, Sec. V-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical evaluation whose main quantitative result is compromised by apparent duplicate rows in Table I. Even after correction, the contribution is limited to the dataset and the comparison, and the citation base is largely documentation-level. I would not recommend acceptance without a corrected, reproducible evaluation; if the duplication reflects the actual collection design, the conclusions may change materially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The ROAD-Almaty dataset is the real contribution here. Collecting and annotating 1,844 images in Almaty across varied weather and lighting is legitimate work, and the paper fills a genuine geographic gap in the driving-scene literature. The evaluation protocol is simple and honestly framed: test three off-the-shelf detectors without retraining. That is a reasonable thing to do, and the authors correctly note the lack of Central Asian benchmarks.\n\nThe numbers, however, do not survive scrutiny. Table I lists Seq1 and Seq4 with identical IoU/F1 values for every model and every threshold, and Seq2 and Seq5 likewise. Figure 2 repeats the same duplicates. If these are meant to be distinct test sequences, exact agreement across all models and thresholds is implausible. If they are the same sequence listed twice, then the reported averages double-count two of the five sequences. Either way, the headline F1-scores and the 46%/27% improvement percentages are not trustworthy as stated. This is a load-bearing flaw: the paper's only quantitative claim is the model ranking, and that ranking rests on these averages.\n\nThe absence of released data, code, or error bars makes the problem worse. Without the dataset, a reader cannot check whether Seq4 and Seq5 are actually different. The test selection procedure is also underspecified—20% of frames held out 'to maximize environmental variability' is not a precise protocol. And the small scale (1,844 images, five sequences) means even corrected results would have wide uncertainty.\n\nWhat the paper does well beyond the data is the framing: it connects the domain-shift literature to a concrete, understudied region and draws a sensible practical conclusion (local data or adaptation is needed). But that conclusion would be the same regardless of which detector wins. The comparison among models, which is the stated focus, is currently unsupported.\n\nI would not cite the performance numbers in their present form. The dataset, if released, would be citable as a resource. The paper deserves a serious referee only if the authors can fix the table and release the data. As submitted, it should not be accepted.","headline":"A genuinely new dataset from an underrepresented region, but the central performance table duplicates sequence rows, so RT-DETR's reported advantage is not established.","tokens_in":8297,"tokens_out":2186,"would_cite":false,"duration_ms":20761,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that RT-DETR, evaluated without retraining on Kazakhstan's ROAD-Almaty dataset, outperforms YOLOv8s and YOLO-NAS, while all three degrade under stricter IoU and harsh weather.","keywords":["domain generalization","object detection","autonomous driving","ROAD-Almaty dataset","YOLOv8s","RT-DETR","YOLO-NAS","domain shift"],"falsifier":"Recompute the average F1 scores after removing the duplicated sequences (keep only one of Seq1/Seq4 and one of Seq2/Seq5) and verify whether RT-DETR still outperforms YOLOv8s and YOLO-NAS by approximately 46% and 27%; a rerun on the original non-duplicated frames, with per-frame scores reported, would settle the ranking.","tokens_in":7293,"feed_emoji":"🚗","tokens_out":9634,"duration_ms":77405,"temperature":0.7,"pith_summary":"The paper tries to establish how well three popular pre-trained object detectors transfer to a driving environment they were not trained on: the streets of Almaty, Kazakhstan. Using a newly built dataset of 1,844 annotated dashcam frames spanning clear, rainy, foggy, and cloudy conditions plus day and night, the authors test YOLOv8s, RT-DETR, and YOLO-NAS with no retraining. They report that RT-DETR reaches an average F1-score of 0.672 at IoU=0.5, ahead of YOLOv8s (0.458) and YOLO-NAS (0.526), and that all models drop roughly 20% when the IoU threshold rises to 0.75. If correct, this makes RT-DETR the safer default among these three for deployment in Central Asian traffic and reinforces the need for geographic diversity in training data.","feed_headline":"RT-DETR leads off-the-shelf detectors on Almaty roads","feed_subtitle":"Off-the-shelf test on 1,844 Almaty frames: RT-DETR scores F1 0.67, beating YOLOv8s and YOLO-NAS without retraining.","key_machinery":"The central object is the ROAD-Almaty test set: 1,844 annotated frames drawn from one-minute dashcam recordings in Almaty at 30 fps, sampled to 10 fps, with about 20% of frames reserved for testing and selected to maximize weather and lighting variability. The comparison mechanism is a zero-retraining protocol in which each pre-trained model is scored on the same five test sequences with COCO-style IoU/F1 metrics at thresholds 0.5 and 0.75, so any performance difference is attributed to the architecture's capacity to generalize rather than to adaptation.","core_discovery":"On the paper's own terms, the discovery is that RT-DETR, a transformer-based detector, generalizes better to the ROAD-Almaty domain than two CNN-based YOLO variants when all three are evaluated without retraining. At IoU=0.5, RT-DETR's average F1 is 0.672 versus 0.458 for YOLOv8s and 0.526 for YOLO-NAS, and the same ordering holds at IoU=0.75 (0.525 versus 0.325 and 0.400). The authors interpret this as evidence that transformer-based feature extraction is more robust to domain shift, while noting that every model degrades under stricter localization and in difficult conditions such as heavy snowfall and low light, which they take as a case for domain adaptation rather than reliance on any single architecture.","pith_inferences":["Because Table I lists identical scores for Seq1 and Seq4 and for Seq2 and Seq5, the effective test size may be three rather than five, and the reported margins may shrink or change on a deduplicated evaluation.","A direct test of the paper's architectural explanation would be to run the same three models on a second, geographically distinct dataset, since the current evidence covers only Almaty.","The paper itself notes that its three model references are official documentation rather than peer-reviewed publications, so reproducing the comparison requires exact checkpoint versions and weight sources.","The paper's own recommendation implies a testable extension: fine-tune each model on a small sample of local frames and measure whether RT-DETR's lead persists after adaptation or whether the YOLO variants catch up."],"forward_implications":["If RT-DETR is indeed the most robust of the three without retraining, teams deploying vehicle detection in Central Asia should prefer it over YOLOv8s and YOLO-NAS until local adaptation data are available.","The roughly 20% F1 drop when IoU moves from 0.5 to 0.75 implies that strict localization requirements will need domain adaptation even for the best-performing model.","The large performance gap in heavy snow and low-light sequences shows that geographic and climatic diversity in training data is a concrete reliability lever for autonomous driving systems.","The ROAD-Almaty dataset can serve as a reusable benchmark for measuring how other pre-trained detectors behave in an underrepresented driving domain."],"supporting_citations":[{"why":"Supplies the premise that dataset bias causes models trained on one distribution to degrade on another, motivating the domain-generalization test.","marker":"[6]"},{"why":"Establishes the expectation that recognition performance falls in novel geographic environments, the phenomenon the paper measures in Almaty.","marker":"[7]"},{"why":"Defines the YOLOv8s model and checkpoint used as one of the three compared detectors.","marker":"[23]"},{"why":"Defines the RT-DETR model and checkpoint whose transformer-based detection is the paper's best performer.","marker":"[24]"},{"why":"Defines the YOLO-NAS model and checkpoint used as the third compared detector.","marker":"[25]"}],"fun_headline_variants":["RT-DETR beats YOLO variants on Almaty roads without retraining","Transformer detector generalizes best on Kazakh driving dataset","Study: RT-DETR tops YOLOv8s and YOLO-NAS on ROAD-Almaty","Off-the-shelf detectors: RT-DETR wins on snowy Almaty roads","Domain gap: RT-DETR outperforms YOLO models on Almaty dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported averages treat the five test sequences as five independent samples of the same target domain, but Table I lists identical scores for Seq1 and Seq4 and for Seq2 and Seq5, so the headline numbers rest on duplicate data.","fun_headline_variants_meta":{"raw":{"variants":["RT-DETR beats YOLO variants on Almaty roads without retraining","Transformer detector generalizes best on Kazakh driving dataset","Study: RT-DETR tops YOLOv8s and YOLO-NAS on ROAD-Almaty","Off-the-shelf detectors: RT-DETR wins on snowy Almaty roads","Domain gap: RT-DETR outperforms YOLO models on Almaty dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1693,"prompt_tokens":971,"completion_tokens":722,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":587,"tokens_out":722,"duration_ms":6337,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:09:18.440132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the average F1 scores after removing the duplicated sequences (keep only one of Seq1/Seq4 and one of Seq2/Seq5) and verify whether RT-DETR still outperforms YOLOv8s and YOLO-NAS by approximately 46% and 27%; a rerun on the original non-duplicated frames, with per-frame scores reported, would settle the ranking.","supporting_citations":[{"cited_title":"Unbiased look at dataset bias,","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that dataset bias causes models trained on one distribution to degrade on another, motivating the domain-generalization test."},{"cited_title":"Recognition in terra incognita,","cited_arxiv_id":null,"evidence_quote":"Establishes the expectation that recognition performance falls in novel geographic environments, the phenomenon the paper measures in Almaty."},{"cited_title":"Yolov8 documentation,","cited_arxiv_id":null,"evidence_quote":"Defines the YOLOv8s model and checkpoint used as one of the three compared detectors."},{"cited_title":"Rt-detr: Real-time detection transformer,","cited_arxiv_id":null,"evidence_quote":"Defines the RT-DETR model and checkpoint whose transformer-based detection is the paper's best performer."},{"cited_title":"Yolo-nas documentation,","cited_arxiv_id":null,"evidence_quote":"Defines the YOLO-NAS model and checkpoint used as the third compared detector."}],"review_version":1}