{"id":"77b84a35-9984-4f09-8a4e-bca05ba96eaf","arxiv_id":"2411.17347","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A NAO robot pipeline using MoveNet-based angle features and a CKCNN whistle detector recognizes referee signals in real time, with strong test numbers but 50% real-match gesture recall.","lead":"This paper reports a real-time system that lets NAO soccer robots recognize a human referee's raised-hands gesture and whistle during RoboCup matches. It combines a two-stage keypoint-and-angle gesture classifier with a continuous-kernel CNN audio detector, and reports test accuracies near 99% with much lower real-match gesture recall.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'improves referee-signal interpretation' claim is not established: real-match gesture recall is 50% on only 18 situations with no baseline or confidence interval.","rationale":"The Reader's verdict is CONDITIONAL and already identifies the same load-bearing assumption: the private-environment gesture model does not transfer to live matches, as evidenced by the 50% real-match recall. My stress-test agrees and adds a statistical and baseline framing: with only 18 real situations, the reported 50% recall has a wide confidence interval, and without a baseline the central 'improves HRI' claim cannot be evaluated. The paper is honest about the limitation in its conclusion, but that does not make the abstract's enhancement claim supported. No internal inconsistency or methodological fraud is alleged; the concern is purely about the evidence for the central claim. Therefore the Reader's CONDITIONAL verdict is appropriate, and no change to the verdict is needed beyond the stated conditions of releasing artifacts and demonstrating that real-match performance is statistically meaningful and better than the status quo.","tokens_in":122,"tokens_out":4604,"duration_ms":112039,"concrete_test":"Run a controlled evaluation on at least 50 real referee signal situations (or bootstrap the existing 18) and compute event-level recall and precision with Clopper-Pearson 95% confidence intervals; compare with a no-gesture baseline and with the existing Game Controller state signal. If the lower confidence bound for gesture recall does not exceed the baseline, the 'improves interpretation' claim is not supported. As a secondary check, log MoveNet keypoint confidence for all missed gestures to confirm whether low-confidence keypoints, rather than classifier error, cause the 50% recall.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4, Table 1b reports gesture recognition Test accuracy 99% but Real accuracy 50% (precision 100%, recall 50%) based on 18 real situations over 8 games, while the abstract and Section 4 claim the pipeline 'enhances real-time human-robot interaction.' In the deployment scenario, half of the referee's standby-to-ready gestures are missed. The paper's explanation that high precision avoids rule penalties does not quantify the benefit: a system that rarely gestures but never false-positives would show high precision and poor recall, and the authors provide no baseline or ablation showing that the 50% recall improves state transitions over current Game Controller behavior. With n=18, the 95% Clopper-Pearson interval for recall 50% is approximately [26%, 74%], so the real-match deficit is not even measured tightly. The private-environment training data described in Section 3.2.1 and the 192x192 crop discussed in Section 3.2 are likely causes of the transfer gap, but the paper does not log keypoint confidence or failure cases to confirm this. Absent artifact release and a statistically grounded event-level evaluation, the strongest claim that the system improves referee interpretation in RoboCup is unsupported even though the whistle component performs better.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript, from the AIRO 2024 workshop, presents a multimodal pipeline for a NAO robot in RoboCup SPL to interpret a human referee through two modalities: whistle detection using Continuous Kernel Convolutional Neural Networks (CKCNNs) and gesture recognition via a two-stage architecture (MoveNet keypoint extraction followed by joint-angle features and a classifier). The authors report a whistle test accuracy of 98.02%, a real-match whistle accuracy of 75% during play and 100% during ready/set phases, and a gesture test accuracy of 99% but a real-match gesture accuracy of only 50%. The paper claims both pipelines run in real time on the NAO (about 0.8 ms for whistle and 200 ms for gesture) and that the system enhances real-time human-robot interaction by reducing reliance on network-based Game Controller messages.","tokens_in":6145,"tokens_out":4626,"duration_ms":45501,"significance":"If the claimed results were fully supported, the work would be a useful contribution to RoboCup SPL, where reducing dependence on the Game Controller is an active goal. The manuscript honestly separates whistle results by game phase, which is a strength, and the real-time numbers, if reproducible, would be relevant to the NAO platform's limited compute. The whistle component appears to perform reasonably well in real matches, especially in quieter phases. However, the central claim that the system improves referee-signal interpretation is not yet established because the gesture subsystem, which is one of two core modalities, has only 50% recall in real matches on a very small sample, and the test-set numbers are reported without statistical grounding, error bars, baselines, or ablations. The paper's own acknowledgment that gesture recognition 'faced challenges in real-world conditions' (Section 5) is consistent with this assessment.","major_comments":[{"comment":"The real-match gesture performance is the main evidence for the paper's central claim of improved referee interpretation, but it is based on only 18 situations and reports a recall of 50%. The 95% Clopper-Pearson interval for 50% recall with n=18 is approximately [26%, 74%], so the true recall is highly uncertain. Moreover, no baseline or ablation is provided to show that a 50% recall improves state transitions over the existing Game Controller behavior. A system that almost never produces a gesture detection would achieve high precision and low recall, so the 100% precision alone does not quantify the practical benefit. The paper needs event-level results with confidence intervals and a comparison against the baseline state-transition mechanism.","section":"Section 4, Table 1b"},{"comment":"The manuscript acknowledges that the 192x192 input and the ROI crop cause detail loss when the referee is distant, and the real-match gesture recall drops from 99% (test) to 50% (real). However, the paper does not log keypoint confidence scores or failure cases to confirm this domain-shift hypothesis, nor does it provide an analysis of the 18 real situations (e.g., distance, occlusion, pose variability). Without such failure analysis or additional real-match data, the applicability of the gesture pipeline under actual RoboCup conditions is not convincingly established, and the reported test accuracy is not a reliable predictor of deployment behavior.","section":"Section 3.2"},{"comment":"The training and test data for both modalities are not described in sufficient detail to assess generalization. For whistle detection, the dataset is cited from [16], but the exact split, the number of samples per condition, and the preprocessing steps (e.g., STFT window size, overlap) are not specified; the reported 147,440 test samples are not characterized in terms of class balance or recording conditions. For gesture recognition, the dataset was 'collected by our team in a private environment' with no sample count, number of participants, pose variations, or cross-validation procedure, and the test set has only 153 samples with no error bars. These missing details make the reported 99% test accuracy unverifiable and the transfer gap hard to interpret.","section":"Sections 3.1.1 and 3.2.1"},{"comment":"The real-time claim central to the title is supported only by the sentence 'Both pipelines are fast enough to run on a NAO robot in about 0.8 ms (whistle) and 200 ms (gesture).' No measurement methodology is given: the specific NAO hardware revision, the number of runs, the variance, whether the gesture time includes the ROI crop and MoveNet inference, and how the 4-consecutive-frame confirmation affects end-to-end latency are all unspecified. Since 'real-time' is a core contribution, the paper should report a reproducible timing benchmark with these details.","section":"Sections 3.1 and 4"}],"minor_comments":[{"comment":"The abstract uses 'CCNNs' while the body uses 'CKCNNs'; please use a single acronym consistently, preferably 'CKCNN' with a definition at first use.","section":"Abstract and Section 3.1"},{"comment":"The angle formula θ = atan2(BCy, BCx) − atan2(BAy, BAx) should state the units and whether the angle is normalized to a fixed range; this matters for the classifier input and for reproducibility.","section":"Section 3.2"},{"comment":"In the whistle results table, the row 'Real (Play)' has a typo in the text 'Lowers precision'; also, reporting the true-positive/false-negative counts for the 73 real situations would make the 75% and 100% accuracy figures easier to interpret.","section":"Section 4, Table 1"},{"comment":"The CKBlock diagram uses an arrow '-> + -> output' that implies a residual connection, but the skip connection is not described in the text; please make the residual addition explicit.","section":"Section 3.1"},{"comment":"The sentence 'the dataset [16] is a collection of audio files' is ambiguous: it is unclear whether the authors reused the dataset of Kleingarn and Brämer or collected their own audio following that reference; please clarify the provenance and availability of the whistle dataset.","section":"Section 3.1.1"},{"comment":"The paper does not state whether code or models will be made available; a data-availability statement would support reproducibility and is especially important given the small real-match evaluation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a workshop report, and the gesture evaluation is currently too thin for a full journal publication. The whistle results and the explicit separation by game phase are promising, and the real-time numbers provide a useful data point, but the central claim of improved HRI needs stronger evidence. I encourage the editor to consider whether the scope of the journal warrants a paper with such limited real-world evaluation; if so, the authors should be asked to address the statistical and baseline issues raised in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about deployed HRI on the NAO. The new thing is not a method - MoveNet, angle features, and CKCNN are all reused - but the integrated two-stage referee-signal pipeline tested during actual RoboCup matches. That integration, plus the explicit split between lab test and real-match numbers, is genuinely useful.\n\nThe whistle side is the stronger half. 98% test accuracy on 147k windows, 75%/100% real accuracy split by game phase, 0.8 ms on the NAO, and the 59.1k-parameter CKCNN design are all concrete. They also honestly separate play (crowd noise) from ready/set, which supports the real-time claim.\n\nThe soft spot is the gesture pipeline. Test accuracy is 99%, but real-match recall is 50% on 18 situations over 8 games. That is not a tight measurement: with n=18, the 95% CI for 50% recall is roughly 26-74%, so the real deficit could be far better or far worse. The paper explains the precision-over-recall choice as avoiding rule penalties, but it provides no baseline or ablation showing that 50% recall actually improves game-state transitions compared to just waiting for the Game Controller message. The authors themselves note the private-environment training data and the downscaled 192x192 crop as likely causes of the transfer gap, and the conclusion admits gesture recognition faced challenges. So the abstract's claim about enhancing real-time human-robot interaction is only partly supported: the whistle half is fine, the gesture half is a demonstration with an unresolved domain gap.\n\nOther soft spots: no code, data, or error bars, and the gesture angle features are standard. None of this is fatal - the claims in the paper are mostly hedged and the worst numbers are in the tables, not hidden. But the central claim about improving referee interpretation needs the gesture gap addressed.\n\nWho is this for? RoboCup SPL teams and anyone working on low-resource referee signal detection. A serious referee should see it: it is an honest engineering paper with real-match evaluation, and the right move is to send it to review with a request for artifacts, more event-level data, and a Game Controller baseline. I would not cite it in my own work yet, but I would read a revised version.","headline":"Honest RoboCup SPL system paper: whistle detection is solid and well-reported, but the gesture claim is undercut by 50% real-match recall on only 18 situations; still worth a referee's time.","tokens_in":6641,"tokens_out":1442,"would_cite":false,"duration_ms":14562,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A NAO robot reads a human referee's gestures and whistle in real time, with the reported pipeline running at about 0.8 ms per whistle window and 200 ms per gesture frame.","keywords":["Human-Robot Interaction","Audio Communication","Gesture Communication","Soccer Robots","Whistle detection","Pose estimation","NAO robot","RoboCup"],"falsifier":"A decisive check would be to run the exact same gesture pipeline on a held-out set of frames recorded during official SPL matches at regulation referee distance, and measure recall; if recall stays near the reported 50% while test-set recall is 99%, the claim that the two-stage keypoint-plus-angle pipeline generalizes to live matches is refuted. For whistle detection, a similarly targeted evaluation would isolate whistle windows during crowd cheering after goals, since the paper already reports 75% accuracy during play and a larger sample would confirm whether the crowd-noise failure is systematic.","tokens_in":1639,"feed_emoji":"🤖","tokens_out":3267,"duration_ms":91348,"temperature":0.7,"pith_summary":"The paper tries to establish that a NAO robot, working under RoboCup's limited-computing and minimal-network rules, can interpret a human referee's signals directly from two modalities: a two-stage gesture pipeline that extracts body keypoints, converts them to joint angles, and classifies the pose, plus a continuous-kernel CNN that detects whistle sounds from short audio windows. If this holds, robots could react to the referee rather than waiting for a network message, which would make game-state transitions faster and more autonomous. Reported test accuracies are high, 98.02% for whistle windows and 99% for gesture samples, but real-match numbers drop to 75% whistle accuracy during play and 50% gesture recall, so the paper's central claim is best read as a feasibility result for real-time multimodal referee perception, with the remaining gap flagged as future work.","feed_headline":"A NAO robot reads referee gestures and whistle in real time","feed_subtitle":"Two lightweight pipelines hit 99 percent gesture and 98 percent whistle accuracy on tests, with real-match gaps.","key_machinery":"Two mechanisms carry the argument. The first is the gesture pipeline: MoveNet, a lightweight keypoint extractor built for real-time pose estimation, turns a 192x192 crop of the referee into skeleton keypoints; the system then computes the angles at the shoulder and elbow for both arms, namely hip-shoulder-elbow and shoulder-elbow-wrist, using $\\operatorname{atan2}$, which gives scale- and rotation-invariant features for a small classifier, and a gesture is accepted only after it appears in four consecutive frames. The second is the whistle detector built from CKBlocks, continuous kernel convolutions in which the convolution kernel is generated by a small MLP that takes the relative position $(t-\\tau)$ as input, allowing a long kernel with only 59.1k total parameters; audio is converted to STFT windows of 513 frequency bins and classified as whistle or no-whistle. Together these mechanisms supply the speed and compactness that make on-robot referee interpretation feasible.","core_discovery":"On the paper's own terms, the central discovery is that a compact continuous-kernel CNN for whistle detection and a two-stage keypoint-plus-angle classifier for gestures can both run on NAO's embedded hardware in real time, with about 0.8 ms per whistle window and 200 ms per gesture frame. In controlled test conditions the gesture classifier reaches 99% accuracy, precision, and recall, and the whistle detector reaches 98.02% accuracy, though with lower precision on the heavily imbalanced whistle data. In eight live RoboCup matches, whistle detection remains 100% accurate in the ready/set states but falls to 75% during play, while gesture recognition keeps 100% precision but drops to 50% recall across 18 real situations. The authors attribute the decline to crowd noise masking whistles and to detail loss when the distant referee is cropped and downscaled, and they identify improved noise resilience and gesture robustness as the open problems.","pith_inferences":["The reported real-match gesture recall of 50% suggests that the crop-and-downscale step is the main bottleneck; a testable extension is to increase input resolution or classify over keypoint sequences rather than single frames.","The paper leaves implicit that the four-consecutive-frames rule adds about 800 ms of latency; a confidence accumulator over frames could cut that delay while preserving precision.","Because the CKCNN decouples kernel size from parameter count, the same whistle architecture could likely be retrained on different sampling rates or feature representations without redesign, an advantage the paper states but does not fully exploit.","A combined referee-interpretation layer that fuses whistle detections with gesture detections and the current game state could recover some of the missed signals in noisy play, but the paper stops short of proposing such a fusion."],"forward_implications":["Game-state transitions can be triggered directly by the referee's raised-hands gesture after four consecutive recognized frames, bypassing the delay of the Game Controller network message.","Whistle detection can run continuously on the NAO at about 0.8 ms per window, making it cheap enough to gate state changes during a match.","The 10:1 class imbalance in the whistle dataset lowers precision on test data, but a consensus approach across multiple robots could suppress false whistle detections in practice.","Because the whistle carries no intrinsic meaning, the detected whistle must be interpreted with the current game context, which the paper notes explicitly as a grounding requirement.","The joint-angle features are rotation- and scale-invariant, so a single trained gesture classifier can in principle work across different referee distances and camera orientations as long as the keypoints are visible."],"supporting_citations":[{"why":"Supplies the continuous kernel convolution method that the whistle detector's CKBlocks are built on.","marker":"[1]"},{"why":"Demonstrates a two-stage keypoint extraction and classification approach for NAO soccer robots that the gesture pipeline directly extends.","marker":"[7]"},{"why":"Provides the whistle audio dataset collected in lab and match conditions that the whistle model is trained and evaluated on.","marker":"[16]"},{"why":"The audio library used to extract STFT features and to align manually labelled whistle events with audio samples.","marker":"[17]"},{"why":"Supplies the MoveNet keypoint extractor used to obtain referee skeletons from the cropped 192x192 images.","marker":"[18]"},{"why":"Describes the lightweight MobileNetV2 backbone underlying the keypoint extractor, which is why the pose stage runs fast enough on the robot.","marker":"[19]"}],"fun_headline_variants":["NAO robot decodes referee gestures and whistle in real time","Two lightweight pipelines let NAO understand a human referee live","Real-time gesture and whistle recognition on NAO, with real-match gaps","NAO referee AI: high lab accuracy, real-match reality check","NAO reads referee cues in 200ms gestures and 0.8ms whistle windows"],"cache_read_input_tokens":8832,"weakest_assumption_plain":"The load-bearing premise is that the body keypoints extracted by MoveNet from a cropped and downscaled 192x192 image of a distant referee retain enough pose information for the joint-angle classifier to generalize from a private close-range training environment to live matches, and the paper's own real-match gesture recall of 50% shows this premise frequently fails.","fun_headline_variants_meta":{"raw":{"variants":["NAO robot decodes referee gestures and whistle in real time","Two lightweight pipelines let NAO understand a human referee live","Real-time gesture and whistle recognition on NAO, with real-match gaps","NAO referee AI: high lab accuracy, real-match reality check","NAO reads referee cues in 200ms gestures and 0.8ms whistle windows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2762,"prompt_tokens":841,"completion_tokens":1921,"prompt_tokens_details":{"cached_tokens":768},"prompt_cache_hit_tokens":768,"prompt_cache_miss_tokens":73,"completion_tokens_details":{"reasoning_tokens":1827}},"tokens_in":73,"tokens_out":1921,"duration_ms":48285,"temperature":1.0,"reasoning_tokens":1827,"cache_read_input_tokens":768,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:12:31.272102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to run the exact same gesture pipeline on a held-out set of frames recorded during official SPL matches at regulation referee distance, and measure recall; if recall stays near the reported 50% while test-set recall is 99%, the claim that the two-stage keypoint-plus-angle pipeline generalizes to live matches is refuted. For whistle detection, a similarly targeted evaluation would isolate whistle windows during crowd cheering after goals, since the paper already reports 75% accuracy during play and a larger sample would confirm whether the crowd-noise failure is systematic.","supporting_citations":[{"cited_title":"Di Giambattista, M","cited_arxiv_id":null,"evidence_quote":"Demonstrates a two-stage keypoint extraction and classification approach for NAO soccer robots that the gesture pipeline directly extends."},{"cited_title":"Kleingarn, D","cited_arxiv_id":null,"evidence_quote":"Provides the whistle audio dataset collected in lab and match conditions that the whistle model is trained and evaluated on."},{"cited_title":"URL: https://blog.tensorflow","cited_arxiv_id":null,"evidence_quote":"Supplies the MoveNet keypoint extractor used to obtain referee skeletons from the cropped 192x192 images."}],"review_version":1}