{"id":"6bd6dad2-ab9b-4c34-bba6-cbb1ba10efae","arxiv_id":"2412.16431","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned YOLOv8 models outperform DETR and DETA on hand detection benchmarks, and a hand-area threshold is proposed to flag video frames for forensic review.","lead":"This paper fine-tunes YOLOv8 and detection transformer models for hand detection on four datasets, including a newly annotated 11k hands dataset, and reports that YOLOv8 beats the transformer models. The best model is then used to flag video frames where a hand occupies a large area, as a triage aid for forensic experts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-generated 11k boxes make the combined-dataset ranking partly circular; independent relabeling is needed to support the claim that YOLOv8x is best.","rationale":"The reader's weakest-assumption analysis correctly identifies the semi-automatic 11k bounding boxes as the load-bearing risk. The paper's headline claim that 'YOLOv8 models outperform DETR and DETA on all datasets' and the specific best-model claim (YOLOv8x on the combined set, AP 0.542) depend on the 11k hands test set and its 1,994 images in the combined test set. Section IV.A states plainly that 8,576 of the final ground-truth boxes are predictions of a YOLOv8n model trained on human boxes only up to 2,500 images. This is a self-training loop with no reported verification step, so the near-perfect 11k results (AP 0.998-0.999) and the combined-set ranking are not fully trustworthy. The concern is not that the authors acted improperly; it is that the evaluation protocol does not currently rule out the alternative explanation that the ranking reflects label-model alignment rather than detection quality. The proposed check — independent human relabeling of a random subset, followed by re-evaluation of all models on that subset and on a revised combined test set — is a direct and feasible way to settle the issue. The EgoHands and Open Images results use external ground truth and therefore provide partial independent support for YOLOv8, so the paper is not without merit; however, those datasets alone do not establish the combined-dataset best-model claim because the combined test set is contaminated by self-generated labels. The forensic-value application is also not quantitatively validated against expert annotations, but that is secondary to the central comparison claim. Since the reader's conditional verdict already requires addressing this weakness, my assessment does not change the verdict: the paper should be conditional pending independent label validation.","tokens_in":13670,"tokens_out":2824,"duration_ms":27496,"concrete_test":"Take a random sample of 300 images from the 8,576 auto-labeled 11k images and have 2-3 independent human annotators draw boxes from scratch in YOLO format without seeing the auto labels. Compute inter-annotator IoU, then re-run all four fine-tuned models (YOLOv8n, YOLOv8x, DETR, DETA) on this sample using human boxes as ground truth. If YOLOv8x's AP advantage over DETR on this subset — and after replacing the 1,994 11k test labels in the combined test set with human labels — shrinks to below 0.01 or reverses, the published ranking is an artifact of self-generated labels; if the advantage persists, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central ranking claim depends on 11k labels created by the authors' YOLOv8n self-training loop (Section IV.A). After human-labeling only 500 images, YOLOv8n predicts boxes for the remaining images in two rounds, and 8,576 of the 11,076 'ground-truth' boxes are YOLOv8n outputs. This affects both training and test: the 11k test partition (1,994 images) uses the same auto-generated labels, and the combined test set (7,788 images) contains those images. Any model whose box distribution resembles YOLOv8's is therefore evaluated against a yardstick made by YOLOv8 itself; DETR/DETA's different assignment and refinement behavior is penalized independently of true hand localization quality. Concretely, YOLOv8x/combined AP 0.542 versus DETR 0.515, and 11k AP 0.998-0.999, could be inflated by label consistency rather than detection accuracy. The paper also reports no inter-annotator agreement or manual verification rate for the auto-labeled portion, so the magnitude of the bias is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares YOLOv8n, YOLOv8x, DETR-ResNet-50, and DETA-Swin-Large on four hand-image datasets (EgoHands, 11k hands, Open Images, and a combined set), reporting that YOLOv8 variants outperform the transformer-based detectors across all benchmarks. As a contribution, the authors provide bounding-box annotations for the 11k hands dataset using a semi-automatic pipeline that starts with 500 manually labeled images and then iteratively uses a fine-tuned YOLOv8n to label the remaining images. The best model, YOLOv8x trained on the combined dataset, is then applied to video frames to identify frames containing large hand regions as having high forensic value.","tokens_in":13906,"tokens_out":2963,"duration_ms":27244,"significance":"If the results are valid, the paper offers a practical system for forensic triage and a useful resource (publicly available bounding boxes for 11k hands). The comparison across multiple detectors on several datasets is a reasonable engineering study. However, the central performance claim depends critically on the quality of the 11k hands labels, and because those labels were generated by one of the evaluated models (YOLOv8n) in a self-training loop, the near-perfect AP values on that dataset and the combined-dataset rankings are not independent evidence of detection quality. The forensic-value application is also presented with only anecdotal examples and no quantitative evaluation. These issues limit the current support for the abstract's claims.","major_comments":[{"comment":"The ground-truth labels for the 11k hands dataset are generated in a semi-automatic loop in which YOLOv8n is fine-tuned on 500 manually labeled images, used to predict labels for the rest, and then re-fine-tuned on 1,000 top-confidence predictions plus 500 manual labels; final labels for 8,576 images are YOLOv8n predictions. These model-generated boxes are then treated as ground truth both for training and for the test partition (Table I: 1,994 test images) used in Table IV and in the combined test set (Table VI). Consequently, the reported AP of 0.998/0.999 for YOLOv8n/YOLOv8x on 11k hands measures consistency with YOLOv8n's own output distribution, not independent localization accuracy, and the combined-dataset ranking (YOLOv8x AP 0.542 vs DETR 0.515 vs DETA 0.414) is potentially inflated by label-style agreement. To support the central claim that YOLOv8 models outperform DETR and DETA, the authors should relabel a random subset (e.g., 500-1000 images) of the 11k hands test partition by human annotators, report inter-annotator agreement, and recompute AP on that held-out human-labeled subset; they should also report the manual verification rate for the auto-generated boxes.","section":"IV.A and V.A.2"},{"comment":"The AP-S and AP-M columns in Table IV contain -1.000 for every method on the 11k hands dataset. Negative average precision is not a defined metric; if these entries are intended to indicate that no small or medium objects exist in the 11k hands images, the table should use a standard placeholder such as \"—\" or \"0\" with a footnote explaining the absence of those object-size categories. As printed, the values are numerically nonsensical and obscure the interpretation of the otherwise near-perfect results.","section":"V.A.2, Table IV"},{"comment":"The forensic-value application is a stated contribution, but the evaluation is anecdotal: the paper reports detected frame counts for two example videos and shows a few stills, but provides no quantitative measure of triage effectiveness, such as precision/recall against a manually labeled set of high-value frames, the number of missed high-value frames, or a comparison with expert manual review time. The claim that the approach \"significantly reduc[es] the time required by forensic experts\" (abstract and Section VI) is therefore not substantiated. A small user study or an evaluation on a labeled video subset with a defined ground truth for high forensic value is needed to support this claim.","section":"V.B"}],"minor_comments":[{"comment":"The text states that YOLOv8n and DETR-ResNet-50 \"each achiev[e] an AP of 0.355\" on Open Images, while the abstract claims YOLOv8 models outperform DETR and DETA on all datasets; since YOLOv8x is the top performer on that dataset, the claim is technically valid, but the wording in Section V.A.3 should be clarified to avoid the impression that every YOLOv8 variant beats every transformer on every metric.","section":"V.A.3"},{"comment":"One video name appears as \"ff fIRxz2-M\" with an extra space, which is likely a typo; the YouTube ID should be verified and the table cleaned.","section":"Table VII"},{"comment":"Reference [32] is cited as a Hugging Face leaderboard space with \"and the Hugging Face Team\" as co-authors; this is not a standard archival reference and should be replaced with a citable publication or a proper URL with a retrieval date.","section":"References [32]"},{"comment":"The intro mentions ethical considerations and states the work does not collect new data, but no explicit data-availability statement for the combined dataset or the code is provided beyond the 11k hands box link; adding a statement about where the trained models and evaluation scripts can be obtained would improve reproducibility.","section":"I"}],"recommendation":"major_revision","confidential_remarks":"The self-training loop for the 11k hands labels is the main concern. The authors do disclose the procedure transparently in Section IV.A, which is good, but they do not acknowledge its circularity when interpreting Table IV and the combined results. This is fixable with a human-labeled validation subset. The EgoHands and Open Images results are independent and appear sound; the combined dataset results are not trustworthy until the 11k portion is externally validated. The forensic-value application needs a quantitative evaluation. Overall, the paper has a useful contribution (released boxes) and a reasonable experimental setup, but the headline claims need stronger support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a competent applied benchmark with one real artifact: public bounding boxes for all 11k hands images. That part is worth having. But the headline claim that YOLOv8 outperforms DETR and DETA on all datasets only holds on the two datasets with human-provided labels. The 11k hands evaluation is against labels the authors' own YOLOv8n produced, so the near-perfect APs on that set are not evidence of detection quality.\n\nWhat is genuinely new: the authors manually labeled 500 images, then iteratively self-trained YOLOv8n to label the remaining 8,576, and they release all boxes. The combined dataset is a useful resource for training hand detectors. On EgoHands and Open Images the benchmark tables look solid, and the YOLOv8 advantage over the transformer models is plausible. One side note: DETA-swin-large's AP of 0.248 on EgoHands, versus DETR's 0.702, suggests it was undertrained or badly configured for that dataset, not that the architecture is inherently worse.\n\nThe load-bearing problem is Section IV.A. The 'ground truth' for 11k hands is 8,576 YOLOv8n predictions plus 2,500 boxes generated by the same self-training loop. The 11k test partition and the combined test set both include these images. So the 11k AP of 0.998-0.999 and the combined ranking that puts YOLOv8x on top are inflated by label consistency, not localization accuracy. This is not a minor caveat; it directly undermines the 'all datasets' claim. The forensic-triage application is also not validated: they threshold bounding-box area and show a few frames from YouTube videos, but there is no comparison against expert annotation, no precision/recall for triage, and no measure of how much expert time is actually saved.\n\nThe good news is that the fix is straightforward. Re-annotate a random subset of 11k hands with human labelers, report inter-annotator agreement, and compare models on that independently labeled subset. Also report variance across training seeds. The authors should either restrict their claims to EgoHands and Open Images or re-evaluate on the new labels.\n\nBottom line: this deserves a serious referee, but with major revision. The annotation resource is citable, and the human-labeled benchmark is useful; the circular 11k evaluation and unvalidated triage claim need to be fixed before the central conclusion is accepted.","headline":"Useful annotation resource and a fair benchmark on human-labeled sets, but the 11k hands evaluation is circular because the 'ground truth' is YOLOv8n's own output.","tokens_in":14424,"tokens_out":1910,"would_cite":true,"duration_ms":18067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned YOLOv8 models detect hands more accurately than DETR and DETA transformer models across four hand image datasets, and the best model, combined with a bounding-box area threshold, can flag high-forensic-value video frames for…","keywords":["hand detection","forensic value","YOLOv8","DETR","DETA","object detection","semi-automatic annotation","video frame triage"],"falsifier":"Manually re-annotate a random sample of roughly 300 images from the 11k hands dataset without reference to the paper's labels, then compute the average precision of the fine-tuned YOLOv8, DETR, and DETA models against these human labels; if the YOLOv8 models' near-perfect scores drop substantially or their ranking against the transformer models changes, the central comparison is not robust to label quality.","tokens_in":13474,"feed_emoji":"🖐️","tokens_out":9323,"duration_ms":65553,"temperature":0.7,"pith_summary":"This paper attempts to establish that fine-tuned YOLOv8 models are the most accurate hand detectors among the tested approaches for forensic triage, beating the transformer-based DETR and DETA models on every dataset. To do this, the authors assemble a combined dataset from EgoHands, the 11k hands dataset, and Open Images, and they contribute bounding-box labels for all 11,076 images in the 11k hands dataset using a semi-automatic process that starts from 500 manually drawn boxes. Their best model, YOLOv8x trained on the combined dataset, reaches an average precision of 0.542 on the combined test set and clearly outperforms earlier YOLOv3/YOLOv4 hand detectors. The payoff is practical: when a detected hand fills a large part of an image or video frame, that frame is flagged as having high forensic value, so forensic experts need only review the flagged frames rather than scan entire videos.","feed_headline":"YOLOv8 beats transformer detectors at finding hands","feed_subtitle":"A combined-dataset YOLOv8x hits AP 0.542 and ranks video frames by hand size for expert review.","key_machinery":"The machinery that carries the comparison is fine-tuned object detection plus a bounding-box-area ranking rule. The detectors are YOLOv8n and YOLOv8x, DETR-ResNet-50, and DETA-Swin-Large, all fine-tuned on the same four training sets and evaluated with the standard AP and AR metrics. For the 11k hands dataset, the authors build labels through a semi-automatic loop: train a YOLOv8n on 500 manually labelled images, use it to predict boxes on the remaining images, add the 1,000 highest-confidence predictions to the training pool, retrain, repeat once, and finally use the model to label the last 8,576 images; those predicted boxes plus the 2,500 seeded boxes become the dataset's ground truth. The forensic application then takes the detector's output boxes, computes hand area as box height times width, and keeps images or frames whose largest hand exceeds a chosen threshold, so the triage decision is a direct function of the detector output.","core_discovery":"The central claim is that YOLOv8n and YOLOv8x, fine-tuned for hand detection, outperform DETR-ResNet-50 and DETA-Swin-Large on all four datasets, with the combined-dataset YOLOv8x as the best overall method (AP 0.542). On EgoHands, YOLOv8n and YOLOv8x reach AP 0.768 and 0.766; on Open Images, YOLOv8x reaches 0.417; on 11k hands, both YOLOv8 variants score near 1.0, but that dataset's labels were generated by the same semi-automatic YOLO loop. The paper further claims that selecting frames by the area of the largest detected hand, using a tunable threshold, successfully identifies high-forensic-value frames from videos and reduces the number of frames an expert must inspect, reporting for example 8 of 680 frames flagged in one video and 41 of 497 in another.","pith_inferences":["A direct extension the authors do not test is using the detected hand crops as inputs to downstream biometric matching (palmprint or vein recognition); the area threshold would then serve as a recall-oriented pre-filter, and the crop quality would matter more than box size.","Because 8,576 of 11,076 labels in the 11k dataset were produced by a YOLOv8n model, a fair robustness check would be to have human annotators re-label a random subset; if IoU between human and model boxes is low, the near-perfect AP scores on that dataset are partly self-confirmation.","The paper implicitly assumes that a larger hand area equals higher forensic value; in practice, a small hand showing a scar, tattoo, or vein pattern could be more valuable than a large blurred hand. A testable variant would collect expert ratings on a sample of frames and measure how well area-based ranking correlates with those ratings.","The threshold used in the example videos differs per video (30,000 vs 50,000 square pixels), so a practical extension would normalise the threshold by image resolution or camera distance; otherwise a global deployment may need per-video calibration."],"forward_implications":["A detector trained on the combined dataset transfers better across hand-image variety: YOLOv8x (combined) scores AP 0.542 on the combined test set versus 0.070 for the same model trained only on EgoHands, showing that training-data breadth matters more than model size alone.","For real forensic video review, setting a hand-area threshold (e.g., 30,000 or 50,000 square pixels) reduces a video to a handful of candidate frames: the paper reports 8 of 680 frames for one example video and 41 of 497 for another.","YOLOv8 models are sufficient for practical deployment: YOLOv8n, the smallest variant, already achieves AP 0.768 on EgoHands and 0.998 on 11k hands, so forensic triage does not require the largest model.","Transformer detectors are not competitive for this task: DETA-Swin-Large, despite 218M parameters, trails YOLOv8n on the combined test set (AP 0.414 vs 0.499) and performs especially poorly on EgoHands (AP 0.248)."],"supporting_citations":[{"why":"Defines the YOLOv8 architecture whose nano and extra-large variants are fine-tuned and compared.","marker":"[25]"},{"why":"Defines the DETR transformer detector that serves as the main transformer baseline.","marker":"[14]"},{"why":"Defines the DETA transformer detector that serves as the second transformer baseline.","marker":"[15]"},{"why":"Supplies the 11k hands dataset, for which the authors create semi-automatic bounding-box labels.","marker":"[16]"},{"why":"Supplies the EgoHands dataset with its ground-truth hand boxes used for training and testing.","marker":"[29]"},{"why":"Supplies the Open Images hand subset used as the largest training and test source.","marker":"[30]"},{"why":"Supplies the pre-trained YOLOv3/YOLOv4 hand detection baseline that the YOLOv8 results are measured against.","marker":"[34]"},{"why":"Defines the AP and AR metrics used in the evaluation tables.","marker":"[32]"}],"fun_headline_variants":["YOLOv8 beats DETR and DETA across all hand datasets","YOLOv8 top hand detector, transformer models fall short","YOLOv8 finds forensic hands faster than transformer rivals","YOLOv8 outperforms vision transformers on hand detection","Hand-spotting: YOLOv8 wins over DETR and DETA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the semi-automatically generated boxes for the 11k hands dataset are valid ground truth, even though most of them were produced by a YOLOv8n model of the same family that is later evaluated against them.","fun_headline_variants_meta":{"raw":{"variants":["YOLOv8 beats DETR and DETA across all hand datasets","YOLOv8 top hand detector, transformer models fall short","YOLOv8 finds forensic hands faster than transformer rivals","YOLOv8 outperforms vision transformers on hand detection","Hand-spotting: YOLOv8 wins over DETR and DETA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3135,"prompt_tokens":1043,"completion_tokens":2092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":2000}},"tokens_in":659,"tokens_out":2092,"duration_ms":12289,"temperature":1.0,"reasoning_tokens":2000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:34:44.761780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually re-annotate a random sample of roughly 300 images from the 11k hands dataset without reference to the paper's labels, then compute the average precision of the fine-tuned YOLOv8, DETR, and DETA models against these human labels; if the YOLOv8 models' near-perfect scores drop substantially or their ranking against the transformer models changes, the central comparison is not robust to label quality.","supporting_citations":[{"cited_title":"11k hands: Gender recognition and biometric identification using a large dataset of hand images,","cited_arxiv_id":null,"evidence_quote":"Supplies the 11k hands dataset, for which the authors create semi-automatic bounding-box labels."},{"cited_title":"Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions,","cited_arxiv_id":null,"evidence_quote":"Supplies the EgoHands dataset with its ground-truth hand boxes used for training and testing."},{"cited_title":"The Open Images Dataset V4: Unified image classification, object detec- tion, and visual relationship detection at scale,","cited_arxiv_id":null,"evidence_quote":"Supplies the Open Images hand subset used as the largest training and test source."},{"cited_title":"A pre-trained YOLO based hand detection network,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained YOLOv3/YOLOv4 hand detection baseline that the YOLOv8 results are measured against."},{"cited_title":"Open object de- tection leaderboard on Hugging Face,","cited_arxiv_id":null,"evidence_quote":"Defines the AP and AR metrics used in the evaluation tables."}],"review_version":1}