{"id":"efb3b0e4-0da7-462e-9ea0-b206f3bea4ec","arxiv_id":"2411.18164","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new crowd-head detection benchmark for railway platforms and event entrances, with 109,913 annotated heads across 1,886 images, shows YOLOv9 and RT-DETR reach about 91% mAP while models trained on existing datasets fall below 57%.","lead":"This paper introduces a new annotated dataset of 109,913 pedestrian heads in 1,886 images from railway platforms and event entrances, and benchmarks eight object detection models on it. The best models reach about 91% mAP, while models trained on existing public head datasets transfer poorly to these settings, suggesting the new data fills a real gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test block split may leak near-duplicate frames from the same videos into both sets, inflating RPEE-trained mAP (~90%) over cross-domain baselines (~57%).","rationale":"The reader's verdict of CONDITIONAL is appropriate, as is the concern about missing annotation-quality analysis. However, the single most load-bearing threat to the central claim is not annotation noise but the evaluation protocol. The claim that RPEE-trained models outperform public-dataset models is supported by comparing models trained on RPEE and evaluated on an RPEE test set against models trained on other datasets and evaluated on that same test set. This comparison is only meaningful if the RPEE test set is not implicitly familiar to the RPEE-trained models through near-duplicate frames from the same static-camera videos. The split description in Section 3.2 does not rule out temporal leakage: blocks are randomly assigned from within each video, and several sources have frame intervals of 3 s. If a test block is adjacent to a training block, the test frames can be nearly identical to training frames. This would inflate the reported 90.7–90.8% mAP and the gap to the ≤56.9% cross-domain baselines. The proposed temporal split test would settle whether the gap persists when the RPEE test set is strictly held out at the video or experiment level. If the gap largely disappears, the benchmark's value as a specialized training resource is still plausible, but the paper's central comparative claim would need substantial qualification. Because this can be checked and fixed without discarding the dataset contribution, the verdict remains CONDITIONAL rather than moving to ACCEPT or REJECT.","tokens_in":16644,"tokens_out":4109,"duration_ms":41711,"concrete_test":"Re-run the Section 4.4 transfer experiment with a strict video-level temporal split: hold out entire video recordings (or, for the multi-experiment event-entrance data, entire experiments) for testing, and train only on the remaining videos, ensuring no test frame is within 60 s of any training frame. Report mAP for YOLOv9-E, YOLOv8x, and RT-DETR trained on RPEE and on each public dataset under this split. If RPEE-trained mAP drops materially (e.g., by more than 10 points), the 90.7/90.8 vs ≤56.9 gap is substantially explained by train/test similarity rather than dataset quality. Also report the minimum temporal gap between each test frame and its nearest training frame, and the nearest-neighbor image similarity across the split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 describes a block-wise random 70/15/15 split, but blocks are drawn from the same 66 videos, with static cameras and, for the Event Entrances Experiments, frame intervals of only 3 s (Table 2). If blocks from one video are assigned to both training and test—the paper gives no evidence that they are not—the RPEE test set can contain frames only 3 s (or a few block lengths) apart from training frames of the same scene, with nearly identical backgrounds and pedestrian configurations. A model trained on RPEE can exploit this near-duplicate appearance, whereas models trained on FDST, SCUT-HEAD, or Hollywood Heads cannot. The headline comparison in Section 4.4 ('mAP 90.7/90.8 vs at most 56.9', Fig. 8) would then overstate the dataset's contribution to the performance gap. Annotation noise, the reader's concern, affects absolute mAP values, but the split confound directly biases the comparative claim. A secondary internal inconsistency ('heads smaller than 62 pixel2' vs 'less than 36 square pixels' in the Section 4.4 summary) should also be fixed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RPEE-Heads, a pedestrian-head detection benchmark with 109,913 bounding boxes across 1,886 frames from 66 railway-platform and event-entrance videos. It trains eight object detectors (Fast/Faster/Cascade R-CNN, RetinaNet-101, YOLOv7x/8x/9-E, RT-DETR) on this dataset and reports that YOLOv9-E and RT-DETR achieve the highest mAP@0.5 (90.7% and 90.8%). It then trains the top three detectors on public head-detection datasets (FDST, SCUT-HEAD Parts A/B, Hollywood Heads) and evaluates them on the RPEE-Heads test set, reporting substantially lower performance (mAP at most 56.9%). A further analysis relates head size to detection accuracy. The authors conclude that RPEE-Heads is necessary for accurate head detection in railway-platform and event-entrance scenarios and release the dataset and pretrained models at a DOI.","tokens_in":16829,"tokens_out":5255,"duration_ms":50424,"significance":"If the claims hold, RPEE-Heads addresses a real gap: dense-crowd railway and event-entrance scenes are underrepresented in public head-detection benchmarks, and the paper provides a new annotated resource plus a documented comparison of eight detectors. The external-anchor design of the cross-dataset experiment is a strength: models trained on public data and evaluated on RPEE-Heads test a genuine generalization question, and the fixed hyperparameter table makes the within-dataset comparison reproducible in principle. The public release of the data and models is another concrete contribution. However, the significance is conditional on the benchmark-construction evidence: the train/test split procedure and the annotation quality are not yet documented at the level a benchmark paper requires, and the headline performance gap could be materially affected by split leakage. The head-size analysis is correlational and useful, but it is undermined by an internal threshold inconsistency.","major_comments":[{"comment":"The block-wise random split described in Section 3.2 does not prevent the same video (or the same static scene) from contributing blocks to both the training and test sets. This risk is concrete: for the Event Entrances Experiments, frames are sampled every 3 seconds (Table 2), so frames in different sets can be only seconds apart, with nearly identical backgrounds and pedestrian configurations. The paper provides no evidence that blocks from a single video are constrained to a single split. Because the paper's central claim is the large mAP gap between RPEE-trained models (about 90.7-90.8%) and public-dataset-trained models (at most 56.9%, Fig. 8), such near-duplicate leakage would directly bias the comparison in favor of RPEE-trained models. The authors should report the block-to-split assignment at the level of each video, re-run the comparison with a video- or scene-exclusive split, or provide quantitative evidence (e.g., frame-similarity distributions) that training and test frames do not share near-duplicate content.","section":"Section 3.2, Table 2, Section 4.4"},{"comment":"The annotation protocol is described only as 'accurate annotations were manually added' by two annotators; the paper reports no inter-annotator agreement, no quality-control protocol, and no error analysis for the 109,913 bounding boxes. With an average of 56.2 heads per image in dense crowds, annotation omissions and box misplacements are plausible and would directly affect every reported mAP and the benchmark's utility for future work. The authors should report an inter-annotator agreement measure (e.g., per-image IoU or box-level agreement on a random subset) and an error analysis stratified by density, head size, and occlusion, or explain why such verification is unnecessary for this dataset.","section":"Section 3.2, Section 5"},{"comment":"All results are single-run point estimates with no error bars, confidence intervals, or significance tests. The claimed superiority of RT-DETR over YOLOv9-E (90.8% vs 90.7% mAP) is within plausible run-to-run variation, especially for stochastic training; similarly, the cross-dataset gap in Fig. 8 has no variance estimate. The authors should provide at least repeated runs (or a standard error) for the top models and the main cross-dataset comparison, or otherwise temper claims of 'outperform' and 'significant' where the differences are small.","section":"Section 4.2, Table 5, Section 4.4"},{"comment":"The head-size conclusion is internally inconsistent: the summary states that 'objects or heads occupying less than 36 square pixels' are the problem, while the analysis and Table 6 consistently use a threshold of 'less than 62 pixel2' (and the Conclusion says 'within 62 pixel2'). This is not a formatting issue: the threshold is load-bearing for the paper's third contribution. The authors should correct the threshold and ensure that the text, figures, and tables use the same head-size categories.","section":"Section 4.4, Table 6, Section 5"}],"minor_comments":[{"comment":"The dataset name is written as 'RPEE-Head dataset' in one place; it should be 'RPEE-Heads' for consistency.","section":"Section 3.2"},{"comment":"'SCUT-HEAD Par B' is a typo; it should be 'SCUT-HEAD Part B'.","section":"Section 4.4"},{"comment":"The head-size bins are not contiguous: '02-62' and '72-122' leave the range 63-71 pixel2 unaccounted for, as do the later ranges. Please make the bins exhaustive and non-overlapping, or state explicitly why intermediate sizes are excluded.","section":"Figure 4, Table 6"},{"comment":"The random 70/15/15 split is described as 'approximately and randomly' performed, with no random seed or split file referenced. For reproducibility, the exact split (or a seed) should be provided with the dataset.","section":"Section 3.2"},{"comment":"The abbreviation 'YOLOV8x' appears without a hyphen in several places; it should be 'YOLOv8x' consistently for readability.","section":"Throughout"},{"comment":"Reference [45] contains 'iEEE' and should read 'IEEE'; also, the citation formatting in the reference list is inconsistent in a few entries.","section":"References"},{"comment":"The paper does not explicitly state how the public-dataset training splits were chosen and whether the same hyperparameters and input sizes from Table 4 were used for the FDST/SCUT-HEAD/Hollywood models. Please clarify this protocol, since it affects the interpretation of Table 7 and Fig. 8.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is potentially valuable and within the scope of the journal, and the cross-dataset experiment is a good external anchor. However, the split-leakage concern is substantive: with 3-second frame intervals and block-wise random splitting, the headline generalization claim could be an artifact of near-duplicate training/test frames. This is fixable by re-splitting at the video level and rerunning the main comparison, so I do not recommend rejection at this stage, but the revision must provide the split evidence and correct the annotation-quality documentation before the benchmarking claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The RPEE-Heads dataset is a real contribution: 109,913 manually annotated heads in 1,886 images from railway platforms and event entrances, a domain that indeed lacks bounding-box resources. The data are public, the annotation effort is substantial, and the transfer experiment is the right idea—training on FDST, SCUT-HEAD, and Hollywood Heads and testing on RPEE gives an externally anchored measure of domain shift. The observed gap (at most 57% mAP for public-trained models vs. ~90% for RPEE-trained) makes the case that these scenes need dedicated data.\n\nThe soft spots are real, though. The main one is the train/test split. The paper divides each video into blocks and then randomly assigns blocks to train, validation, and test. Since the event-entrance experiment videos were sampled every 3 seconds from static cameras, blocks from the same video in both train and test can contain near-duplicate frames with almost identical backgrounds and pedestrian layouts. The paper gives no evidence that blocks were kept together by video. If they weren't, the RPEE-trained models' mAP is inflated by appearance leakage, and the 90 vs. 57 gap overstates the dataset's contribution. This does not destroy the central claim—public-trained models still clearly transfer poorly to these scenes—but it undermines the precise magnitude.\n\nTwo smaller issues: there is no inter-annotator agreement or annotation quality check, and all metrics come from single runs with no error bars. The head-size analysis has an internal inconsistency (\"less than 62 square pixels\" in one place vs. \"less than 36 square pixels\" in the summary) and the size bins have gaps (02–62, 72–122, ...), which should be cleaned up.\n\nWorth a serious referee, but it needs major revision: redo the split at video level or show that same-video leakage cannot happen, add annotation agreement and multiple seeds, and fix the head-size reporting. I'd bring it to reading group, and if you work on crowd analysis, it's citable as a dataset resource regardless of the leak fix.","headline":"A genuinely useful crowd-head dataset, but the headline accuracy gap is likely inflated by a block-level train/test split that leaks near-duplicate frames from the same videos.","tokens_in":17369,"tokens_out":1824,"would_cite":true,"duration_ms":19315,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new dataset of 109,913 pedestrian-head boxes closes a domain gap in head detection at railway platforms and event entrances, raising mAP from at most 56.9% to 90.8%.","keywords":["RPEE-Heads","pedestrian head detection","crowd analysis","object detection benchmark","railway platforms","event entrances","head-size analysis","RT-DETR"],"falsifier":"Re-annotate a random sample of, say, 200 RPEE-Heads test frames with independent annotators and measure per-box Intersection-over-Union agreement; if agreement falls well below the 0.5 IoU threshold used to score detectors, the reported 90%+ mAP may partly reflect lenient ground truth. A sharper test: have one experienced annotator re-label the entire test set and recompute YOLOv9-E and RT-DETR mAP; a large drop would indicate the original labels carried the headline numbers.","tokens_in":16437,"feed_emoji":"🚆","tokens_out":11728,"duration_ms":93111,"temperature":0.7,"pith_summary":"The paper introduces RPEE-Heads, a benchmark of 109,913 manually annotated pedestrian-head bounding boxes in 1,886 images sampled from 66 videos of railway platforms and event entrances. Its central claim is that this setting is underrepresented in public head-detection datasets, and that the gap shows up as a large accuracy drop: models trained on FDST, SCUT-HEAD, or Hollywood Heads and tested on RPEE-Heads reach at most 56.9% mean average precision, whereas YOLOv9-E and RT-DETR trained on RPEE-Heads reach 90.7% and 90.8%. The paper also reports an empirical size effect: datasets dominated by heads below about 62 square pixels, such as JHU-CROWD++ and NWPU-Crowd, yield poor detector accuracy. If the claims hold, the dataset gives crowd-safety researchers a domain-specific training and evaluation resource, plus a baseline across eight modern detectors.","feed_headline":"Head detection at stations jumps from 57% to 91% mAP","feed_subtitle":"A 109,913-head dataset from railway platforms and event entrances lets real-time detectors top 90%.","key_machinery":"The central object is the RPEE-Heads dataset itself: 1,886 high-resolution frames with normalized bounding-box annotations (class, center x, center y, width, height) for 109,913 visible heads, split by video blocks into 70/15/15 train/validation/test sets to minimize near-duplicate frames between sets. The benchmark machinery also includes five head-size categories (under 62, 72 to 122, 132 to 242, 252 to 322, and above 332 square pixels) and the standard mAP@0.5 metric, which together let the paper attribute performance differences to head scale and to domain mismatch.","core_discovery":"RPEE-Heads is built from 66 video recordings: 15 railway-platform videos, 34 music-concert entrance videos, and 17 indoor event-entrance experiments sourced from an open archive. Frames were sampled at intervals chosen to reduce duplicate scenes, and two annotators manually placed bounding boxes around visible head regions, yielding 109,913 boxes over 1,886 frames with an average of 56.2 heads per frame. The paper evaluates Fast R-CNN, Cascade R-CNN, RetinaNet-101, Faster R-CNN, RT-DETR, YOLOv7x, YOLOv8x, and YOLOv9-E at an IoU threshold of 0.5; YOLOv9-E and RT-DETR score 90.7% and 90.8% mAP with 11 ms and 14 ms inference times, respectively. When the top three models are trained from scratch on FDST, SCUT-HEAD Part A, SCUT-HEAD Part B, and Hollywood Heads and evaluated on the RPEE-Heads test set, their mAP falls to at most 56.9%. The authors conclude that specialized datasets like RPEE-Heads are needed for head detection at railway platforms and event entrances, and that small heads, particularly below 62 square pixels, degrade detector performance.","pith_inferences":["The reported cross-dataset gap is likely a mixture of head-size distribution and viewpoint/background shift; a controlled experiment that matches head-size distributions across datasets while varying camera angle could separate the two effects.","Because only visible head regions are annotated and the source videos exist, a natural extension the paper leaves implicit is using temporal continuity from the videos for occlusion-aware tracking, but the released benchmark does not provide track IDs.","The small-head result points to overhead, low-resolution cameras as the hard regime; one testable next step is combining RPEE-Heads with super-resolution or scale-adaptive detection heads and measuring mAP for the sub-62-square-pixel bin.","If inter-annotator agreement turns out high, the benchmark could also serve as a pretraining set for other dense-crowd domains, not just railway stations and event entrances."],"forward_implications":["Crowd-management systems at railway stations and event entrances can use RPEE-Heads-trained YOLOv9-E or RT-DETR as a real-time head detector with around 90% mAP and 11–14 ms per frame.","The paper gives the community a fixed evaluation suite: eight detectors scored on the same test split with precision, recall, F1, mAP, true/false positives, false negatives, and inference time.","The head-size distribution tables let curators check whether a new dataset is dominated by heads below about 62 square pixels, the range the paper finds most damaging to accuracy.","Block-wise train/validation/test splits across videos mean the reported scores are less likely to be inflated by near-duplicate frames between training and test.","The public release of the labeled data and pretrained models allows the results to be reproduced and extended."],"supporting_citations":[{"why":"YOLOv9-E is the architecture that reaches 90.7% mAP on the new dataset, one of the two headline results.","marker":"[20]"},{"why":"RT-DETR reaches the top 90.8% mAP at 14 ms inference time, the other headline result.","marker":"[34]"},{"why":"YOLOv8x is one of the three models retrained on public datasets for the cross-dataset comparison.","marker":"[19]"},{"why":"SCUT-HEAD is a classroom head dataset whose models drop to 19.7% mAP on RPEE-Heads test images.","marker":"[25]"},{"why":"Hollywood Heads is the public dataset whose best transferred model reaches only 56.9% mAP, the strongest outside baseline.","marker":"[26]"},{"why":"FDST is another public training set used in the cross-dataset comparison, with best transferred mAP 43.4%.","marker":"[42]"},{"why":"NWPU-Crowd's 74.4% share of sub-62-pixel heads supports the paper's small-head claim.","marker":"[27]"},{"why":"JHU-CROWD++'s 61.37% share of sub-62-pixel heads and 22.1% best mAP ground the small-head finding.","marker":"[28]"},{"why":"Supplies 49 of the 66 video recordings covering railway platforms and concert entrances.","marker":"[46]"},{"why":"Supplies the 17 event-entrance experiment videos from an open archive.","marker":"[47]"}],"fun_headline_variants":["Crowd head detection soars to 91% mAP with new RPEE-Heads dataset","New dataset: 109,913 heads for crowd detection hits 91% mAP","Stations and events: specialized head detection dataset reaches 91% mAP","RPEE-Heads: new benchmark for crowd head detection, 90%+ mAP","Why generic datasets fail at crowd heads: RPEE-Heads shows 91% mAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value rests on the assumption that the 109,913 manually placed bounding boxes are accurate and consistent; the paper reports no inter-annotator agreement or error analysis, so if the ground truth is noisy, the reported mAP figures and cross-dataset comparisons are not reliable.","fun_headline_variants_meta":{"raw":{"variants":["Crowd head detection soars to 91% mAP with new RPEE-Heads dataset","New dataset: 109,913 heads for crowd detection hits 91% mAP","Stations and events: specialized head detection dataset reaches 91% mAP","RPEE-Heads: new benchmark for crowd head detection, 90%+ mAP","Why generic datasets fail at crowd heads: RPEE-Heads shows 91% mAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3987,"prompt_tokens":1082,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":2791}},"tokens_in":698,"tokens_out":2905,"duration_ms":18568,"temperature":1.0,"reasoning_tokens":2791,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:26:22.297181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of, say, 200 RPEE-Heads test frames with independent annotators and measure per-box Intersection-over-Union agreement; if agreement falls well below the 0.5 IoU threshold used to score detectors, the reported 90%+ mAP may partly reflect lenient ground truth. A sharper test: have one experienced annotator re-label the entire test set and recompute YOLOv9-E and RT-DETR mAP; a large drop would indicate the original labels carried the headline numbers.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YOLOv8x is one of the three models retrained on public datasets for the cross-dataset comparison."},{"cited_title":"Detecting Heads using Feature Refine Net and Cascaded Multi-Scale Architecture","cited_arxiv_id":"1803.09256","evidence_quote":"SCUT-HEAD is a classroom head dataset whose models drop to 19.7% mAP on RPEE-Heads test images."},{"cited_title":"Context-aware CNNs for person head detection","cited_arxiv_id":null,"evidence_quote":"Hollywood Heads is the public dataset whose best transferred model reaches only 56.9% mAP, the strongest outside baseline."},{"cited_title":"Locality-constrained spatial transformer network for video crowd counting","cited_arxiv_id":null,"evidence_quote":"FDST is another public training set used in the cross-dataset comparison, with best transferred mAP 43.4%."},{"cited_title":"Nwpu-crowd: A large-scale benchmark for crowd counting and localization","cited_arxiv_id":null,"evidence_quote":"NWPU-Crowd's 74.4% share of sub-62-pixel heads supports the paper's small-head claim."},{"cited_title":"Sindagi, Rajeev Yasarla, and Vishal M","cited_arxiv_id":null,"evidence_quote":"JHU-CROWD++'s 61.37% share of sub-62-pixel heads and 22.1% best mAP ground the small-head finding."},{"cited_title":"Crowd management in transport infrastructures (project number 13n14530 to 13n14533)","cited_arxiv_id":null,"evidence_quote":"Supplies 49 of the 66 video recordings covering railway platforms and concert entrances."},{"cited_title":"https://doi.org/10.34735/ped.da, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the 17 event-entrance experiment videos from an open archive."}],"review_version":1}