{"id":"a9dbc352-057c-4b1c-ad34-06b801023cb0","arxiv_id":"2602.01515","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A simulation-trained recurrent model detects out-of-distribution states on a real humanoid at 50 Hz and uses gradient saliency plus an LLM to diagnose failure causes.","lead":"RAPT is a 50 Hz watchdog that learns what normal humanoid-robot behavior looks like in simulation, then flags when the real robot wanders outside that learned pattern and triggers a safe stop or fall. It also turns the robot's own sensor signals into a short language-model explanation of what went wrong, which could make sim-to-real debugging faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hardware false-positive claim is not established: max+kσ calibration (Eqs. 6–8) is not a calibrated low-FPR rule and only 3 nominal trials back the 0.5% FPR operating point.","rationale":"The most load-bearing part of RAPT's central claim is the transfer from simulation-trained thresholds to hardware with 'few false positives.' In simulation, the 4096-environment protocol and five-seed averaging provide reasonable evidence that RAPT outperforms baselines under the paper's operating-point definition. However, the hardware claim rests on the calibration rule of Eqs. 6–8, which is an ad hoc max+kσ procedure rather than a calibrated quantile or conformal method. Because per-dimension NLL distributions are heavy-tailed and ~140 dimensions are thresholded, the max over a short calibration window is a high-variance order statistic; the chosen k does not control the family-wise error rate. The paper's own Discussion concedes the calibration 'does not explicitly model the nominal distribution of each observation dimension' and may cause false negatives. The empirical support for low real-world FPR is just 3 nominal trials, which is statistically uninformative. The abstract/body inconsistencies (89% TPR across 78 trials vs 75% on 24/27 trials; '37% improvement' vs +0.34 absolute Safety Score) make the headline magnitude untrustworthy. These issues do not invalidate the architecture or the simulation results; they mean the central sim-to-real claim is currently conditional on evidence not yet provided. Hence the reader's CONDITIONAL verdict should stand.","tokens_in":12110,"tokens_out":10758,"duration_ms":111436,"concrete_test":"Run the calibration-and-test protocol on the real G1 (or a high-fidelity simulator with matched sensor noise) with 10 different calibration episodes (varying duration and seed), then evaluate on 50+ held-out nominal episodes. Record the max+5σ thresholds and the empirical episodic FPR. If thresholds vary by >10% in NLL units across calibrations, or if any calibration yields an FPR above 0.5%, the max+kσ rule is not stable enough to support the central claim. Alternatively, derive the order-statistic distribution of the per-dimension max over T_cal under a heavy-tailed model and compute the implied per-timestep false-positive probability; check whether it is consistent with the reported 0.5% episode-level FPR.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RAPT maintains low false positives at 0.5% episode-level FPR on hardware depends entirely on the calibration thresholds in Eqs. 6–8. These thresholds are order statistics (max over a brief calibration period) plus kσ, with k=5 per dimension and k=3 on the global mean. This is not a statistically calibrated quantile procedure: for heavy-tailed per-dimension NLL (the paper itself cites joint velocities as heavy-tailed), the maximum is unstable across calibration seeds, and the 5σ margin does not control the family-wise error rate over ~140 dimensions. The result is that the threshold's FPR is not controlled at 0.5%; it is whatever the max-plus-kσ rule produces. On real hardware, only 3 nominal trials were used to report zero false positives; with n=3, a true FPR of 50% would still yield zero observed FPs with probability 12.5%. Thus the real-world transfer evidence for the central low-FPR claim is statistically insufficient, and the Discussion's admission that the calibration strategy 'does not explicitly model the nominal distribution of each observation dimension' and 'may lead to false negatives' underscores that this is the weakest link. The abstract's '89% TPR across 78 trials' also conflicts with Table III's 18/24 = 75% on 27 trials, further reducing confidence in the headline numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RAPT is a self-supervised, 50 Hz deployment monitor for humanoid robots. It learns a recurrent probabilistic reconstruction model from nominal simulation data and detects OOD states using per-dimension NLL thresholds (max + kσ) plus a deterministic range gate. The paper further proposes a post-hoc diagnostic pipeline that combines BPTT-based temporal saliency with an LLM for semantic failure classification. The method is evaluated on four Isaac Lab tasks (4096 environments, 5 seeds, 15 OOD categories) against several baselines, and on a Unitree G1 in 27 real-world trials. The central claims are a 37% TPR improvement over the strongest baseline at 0.5% episode-level FPR in simulation, an 89% TPR across 78 hardware trials (abstract) or 75% recall on 24 anomalous trials (Table III), and 75% root-cause diagnosis accuracy on 16 real failures.","tokens_in":12434,"tokens_out":4964,"duration_ms":53407,"significance":"If the results hold, RAPT addresses a practically important gap: detecting sim-to-real silent failures at high control frequency with low false-positive rates, and providing interpretable failure feedback. The simulation protocol is a clear strength — 4096 parallel environments, 5 seeds, 15 OOD categories, and comparisons to four baselines on identical logs. The public release of code, datasets, and deployment code is also valuable. However, the headline quantitative claims are internally inconsistent: the abstract's 89% TPR/78 trials is not supported by the body's 24 anomalous trials/27 total trials, and the '37% improvement' is not traceable to the reported Safety Scores. Moreover, the max+kσ calibration does not provide a statistical guarantee at the advertised 0.5% FPR, and the hardware false-positive evidence rests on only 3 nominal trials. These are load-bearing issues for the central contribution; the underlying method is plausible but the current manuscript overstates the strength of the evidence.","major_comments":[{"comment":"The numbers do not agree. The abstract states '89% TPR across 78 trials', but §IV.D reports 24 OOD trials, and Table III reports N=27 total trials (24 anomalous, 3 nominal) with Hybrid recall 18/24 = 75%. The abstract's '37% improvement' is also not traceable: Table I gives RAPT Hybrid Safety Score 0.72 vs LSTM-VAE Hybrid 0.38, an absolute 34 percentage points and a relative ~89% improvement, not 37%. The Introduction's '12.5% TPR improvement' is likewise not consistent with Table III (Hybrid recall 75% vs LSTM 50%, or vs RAPT-only 67%). These inconsistencies affect the paper's headline claims and must be resolved.","section":"Abstract; §IV.D; Table III"},{"comment":"The max+kσ calibration is not a calibrated low-FPR procedure. The thresholds are order statistics (the maximum over a short calibration period) plus kσ, with k=5 per dimension over ~140 dimensions and k=3 on the mean. For heavy-tailed per-dimension NLL (the text itself notes joint velocities are heavy-tailed), the maximum is unstable, and no multiple-testing or tail-probability control is provided. Therefore the 0.5% episode-level FPR operating point is not statistically guaranteed by the method; it is an empirical value for one calibration set. The Discussion (Section VI) explicitly concedes that the calibration 'does not explicitly model the nominal distribution of each observation dimension' and 'may lead to false negatives.' This is load-bearing for the central low-FP claim. Please either provide a calibration scheme with finite-sample FPR control (e.g., conformal or Bonferroni-style","section":"§III.C.1; Eqs. (6)–(8)"},{"comment":"The real-world false-positive evidence is statistically insufficient. The paper reports zero false positives during nominal operation, but this is based on only 3 nominal trials. With n=3, a true per-trial FPR as high as 50% would still yield zero observed false positives with probability 12.5%. Combined with the fact that all experiments are on a single G1 platform over short-term deployments, the headline 'low false positives on hardware' is not established. The Discussion acknowledges 'a modest number of trials' and 'restricted subset of OOD events'; this limitation should be reflected in the abstract and conclusion, not only in the limitations paragraph.","section":"§IV.D; Table III"},{"comment":"The metric 'Safety Score (TPR @ 0.5% FPR)' and 'episode-level FPR' are not defined in the main text. How is an episode defined for FPR computation? Is a false positive counted if any gate fires at any timestep? How is the 0.5% operating point chosen from the scored outputs? Without this, the headline comparison is not reproducible. Please provide explicit definitions and, ideally, the full FPR/TPR curves or the method used to set the threshold to the 0.5% operating point.","section":"§IV.E; Table I"}],"minor_comments":[{"comment":"The '15 OOD categories' are not enumerated in the main text; they are only referenced to the supplementary material. A brief table or list would make the simulation protocol self-contained.","section":"§IV.B"},{"comment":"The Hybrid column for RAPT (0.72) is lower than Model Only (0.75), which is counterintuitive. The text says this is caused by range-detector false positives, but the caption of Table I does not explain it. Please add an explicit note or discuss the mechanism more clearly.","section":"§V.A; Table I"},{"comment":"The terms 'safety margin', 'worst-case margin', and the threshold line are not defined in the figure or caption. Define these quantities and their relation to Eqs. (6)–(8).","section":"Fig. 3"},{"comment":"The '12.5% TPR improvement' in the Introduction is not consistent with any ratio in Table III. Please recalculate and unify the numbers with the abstract and results.","section":"Introduction; Abstract"},{"comment":"Minor typos and formatting: 'V AE' with a stray space, inconsistent dashes in '50Hz' vs '50 Hz', the unitree_rl_lab reference lacks a URL, and some equations (e.g., Eq. (6)) use symbols like σ_global that are not explicitly defined. A careful proofread would help.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong simulation protocol and a useful, practical monitoring system, but the internal numerical inconsistencies and the overclaimed calibration guarantee are significant. In my view, the central method is defensible; the issues are correctable within the manuscript's scope. I would be willing to accept after a revision that corrects the abstract/body discrepancies, clearly defines all metrics, and either adds a finite-sample FPR control method or explicitly reframes the FPR as empirical rather than calibrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible engineering contribution and the simulation study is the best part. The architecture is a recombination of known pieces — residual encoder, GRU bridge, Gaussian decoder, integrated gradients — but the specific combination for 50 Hz humanoid monitoring, the per-dimension max+kσ calibration, and the BPTT saliency for diagnosis are new enough. The sim protocol is genuinely good: 4096 parallel envs, 5 seeds, 15 OOD categories, and the Taguchi ablation gives a real sense of which design choices matter. Code, data, and C++ deployment code are promised, which makes it re-implementable. The real-world detection and diagnosis results are useful as indicative evidence, not proof.\n\nThe soft spots are real. The abstract reports 89% TPR across 78 trials; the body reports 18/24 = 75% on 27 trials. Those don't reconcile. The '37% improvement' over the strongest baseline also doesn't line up with the +0.34 Safety Score in Table I (which is actually an 89% relative increase, not 37%). Either the numbers are defined differently between sections or someone made an editing error — either way the reader can't trust the headline.\n\nThe deeper issue is the calibration rule. Equations 6–8 set thresholds as max-plus-kσ over a brief calibration period. That is not a calibrated low-FPR procedure. For heavy-tailed per-dimension NLL, the max is an unstable order statistic, and k=5 on ~140 dimensions doesn't control family-wise error. So the 0.5% FPR operating point is not actually achieved; it's whatever the rule produces. On hardware, only 3 nominal trials back the zero-false-positive claim. With n=3, a true FPR of 50% still gives a 12.5% chance of seeing zero false positives. The paper's own Discussion admits the calibration strategy doesn't model per-dimension nominal distributions and may cause false negatives. That's the load-bearing weakness.\n\nThe diagnosis evaluation is also on 16 selected logs with no baseline comparison, so 75% top-1 is suggestive but not conclusive. The 21 categories are defined by the authors; no inter-rater or chance-level baseline is reported.\n\nOverall: the method is plausible, the sim work is solid, and the writing is honest about limitations. But the hardware headline is overstated and the calibration claim needs either a statistical fix or a very different reporting frame. This deserves a serious referee, not a desk reject — but the referees should push hard on the calibration analysis and the abstract/body consistency. I'd bring it to a reading group if you work on sim-to-real or robotic OOD, and I'd cite the sim protocol and the method as related work, not as a validated deployment guarantee.","headline":"A solid, re-implementable OOD monitor with a strong simulation study, undermined by mismatched hardware headline numbers and a calibration rule that doesn't deliver the advertised false-positive guarantee.","tokens_in":12958,"tokens_out":3957,"would_cite":true,"duration_ms":33534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A recurrent probabilistic model trained only on simulation data detects safety-relevant out-of-distribution states in a 50 Hz humanoid controller, outperforming prior detectors at a fixed 0.5% episode-level false-positive rate and localizin","keywords":["out-of-distribution detection","sim-to-real transfer","humanoid robots","anomaly detection","recurrent neural network","probabilistic reconstruction","threshold calibration","root-cause diagnosis"],"falsifier":"Run the calibration phase on one surface (e.g., a foam mat) and deploy on a different but nominal surface (e.g., concrete) with the same policy; if the detector triggers a safety stop on many steps of purely nominal walking, the static-shift calibration assumption is refuted. Alternatively, use a second humanoid unit of the same model and see whether thresholds calibrated on one unit transfer to the other without false positives.","tokens_in":11940,"feed_emoji":"🤖","tokens_out":6214,"duration_ms":61663,"temperature":0.7,"pith_summary":"This paper is trying to show that reconstruction likelihood — computed by a recurrent probabilistic model trained on nominal simulation data only — can serve as a calibrated, per-dimension deployment-time signal of sim-to-real mismatch. The authors argue that by thresholding per-dimension negative log-likelihoods with a max-plus-k-sigma calibration rule and adding a deterministic range gate, the proposed detector finds safety-relevant out-of-distribution states at 50 Hz while keeping false positives at the strict level required for real hardware. In large-scale simulation across four humanoid tasks it improves true-positive rate by 37% over the strongest baseline at 0.5% episode-level false-positive rate; on a physical humanoid it detects 18 of 24 anomalous trials with zero false positives in 3 nominal trials. Beyond detection, gradient-based temporal saliency plus a language model yields 75% top-1 semantic root-cause accuracy on 16 real failures using proprioceptive data alone. If correct, this reframes domain shift as an observable, interpretable quantity rather than a binary stop signal, enabling safer deployment and debugging of learned controllers.","feed_headline":"Sim-trained monitor catches humanoid faults at 50 Hz","feed_subtitle":"Per-dimension likelihood and a range gate beat prior detectors at 0.5% false-positive rate and explain failures.","key_machinery":"RAPT (Recurrent Anomaly Probabilistic Trajectory Model): a self-supervised recurrent probabilistic trajectory model. A residual MLP encoder maps each observation to a 256-dimensional latent, a GRU bridges temporal context, and a probabilistic decoder outputs a diagonal Gaussian over observations, trained by maximizing likelihood of nominal simulated data. The anomaly signal is per-dimension negative log-likelihood, calibrated with a max-plus-k-sigma rule (k=5 per dimension, k=3 on the global mean) during a brief real-domain calibration episode, and OR-combined with a deterministic bounding-box range gate. For diagnosis, integrated gradients through time on the NLL produce a spatio-temporal s","core_discovery":"The central claim is that a probabilistic recurrent autoencoder trained only on nominal simulation trajectories, evaluated online via per-dimension Gaussian negative log-likelihood, can detect out-of-distribution states under extremely low false-positive budgets and also localize when and where execution departs from nominal behavior. The method treats reconstruction likelihood not as a binary anomaly flag but as a continuous calibrated measure of sim-to-real gap, adapted to the target domain by a brief calibration run that sets max-plus-k-sigma thresholds on both per-dimension and global mean NLL, plus a hyper-rectangle range gate. On four simulation tasks, the hybrid detector achieves the","pith_inferences":["The brittle point is the calibration episode: the max-plus-k-sigma thresholds are maxima over a single nominal run. Stress-testing by calibrating on one surface and deploying on another nominal surface would reveal whether false positives stay at zero; the paper does not do this, so the transfer guarantee is narrower than stated.","Because RAPT produces per-dimension NLL, the same signal could be used online for policy adaptation or as a feature for model-predictive safety filtering, not just as a binary trigger. The paper mentions policy switching as future work but does not demonstrate it.","The hybrid gate is a logical OR, meaning the range detector's blind spots (anomalies within nominal bounds) are inherited; a learned fusion of the two signals might reduce false positives further while keeping sensitivity."],"forward_implications":["RAPT can serve as an online safety trigger for humanoid controllers: inference runs in 1.63 ms (within the 20 ms control budget), enabling safe stops, controlled falls, and recovery behaviors without modifying the underlying policy.","Because likelihood is decomposed per dimension, RAPT indicates which sensors or joints deviate, narrowing debugging and enabling verification of sim-to-real consistency beyond success metrics.","The calibration procedure removes static domain shift, so a simulation-trained detector transfers to hardware with acceptable false-positive rates after a brief nominal run.","Zero-shot language-model diagnosis using only proprioceptive saliency reaches 75% top-1 accuracy, letting operators distinguish terrain, sensor, and actuator failure modes without retraining or hand-labeled data.","The hybrid detector remains sensitive to anomalies that do not violate explicit state bounds, complementing the range gate and catching silent configuration errors such as under-tuned PD gains."],"fun_headline_variants":["50 Hz monitor catches sim-to-real humanoid faults with fewer false alarms","RAPT: 50 Hz OOD detection localizes and explains humanoid failures","Sim-trained 50 Hz monitor flags OOD states with 37% higher TPR","50 Hz watchdog explains humanoid faults: 75% diagnosis accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The brief real-domain calibration episode is representative of all future nominal operation: the max-plus-k-sigma thresholds are derived from the maxima of those calibration residuals, and the sim-to-real gap is assumed to be mostly static and removable by this calibration.","fun_headline_variants_meta":{"raw":{"variants":["50 Hz monitor catches sim-to-real humanoid faults with fewer false alarms","RAPT: 50 Hz OOD detection localizes and explains humanoid failures","Sim-trained 50 Hz monitor flags OOD states with 37% higher TPR","50 Hz watchdog explains humanoid faults: 75% diagnosis accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001272,"raw_usage":{"total_tokens":5064,"prompt_tokens":789,"completion_tokens":4275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":4192}},"tokens_in":533,"tokens_out":4275,"duration_ms":31301,"temperature":1.0,"reasoning_tokens":4192,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:38:28.627150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the calibration phase on one surface (e.g., a foam mat) and deploy on a different but nominal surface (e.g., concrete) with the same policy; if the detector triggers a safety stop on many steps of purely nominal walking, the static-shift calibration assumption is refuted. Alternatively, use a second humanoid unit of the same model and see whether thresholds calibrated on one unit transfer to the other without false positives.","supporting_citations":[],"review_version":1}