Pith. sign in

REVIEW 2 major objections 6 minor 23 references

Raw crowdsourced webcam video of facial reactions can train end-to-end models that detect robot failures and anticipate bad outcomes better than CNN baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:41 UTC pith:KDGHPTGW

load-bearing objection Solid challenge report that packages raw-video datasets and an anticipatory track; claims match the modest numbers and three teams beat near-chance CNN baselines. the 2 major comments →

arxiv 2607.11570 v1 pith:KDGHPTGW submitted 2026-07-13 cs.RO cs.HC

ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions

Maria Teresa Parreira , Micol Spitale , Maia Stiber , Shiye Cao , Amama Mahmood , Chien-Ming Huang , Hatice Gunes , Wendy Ju This is my paper
classification cs.RO cs.HC
keywords Robot FailureError DetectionHuman-Robot InteractionMultimodal InteractionBenchmarkingAnticipation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports the ERR@HRI 3.0 challenge, which released two complementary raw-video datasets so researchers can learn error-related social signals directly from pixels rather than hand-crafted features. The BAD set records spontaneous bystander faces while people watch robot and human failures; the Bad Idea set records anticipatory faces while people predict whether cut-off action clips will end well or poorly. Both were collected via crowdsourcing under everyday lighting, angles, and backgrounds, then split subject-independently so models must generalize to unseen people. Three teams submitted valid models; all of them beat the provided convolutional baselines on the held-out tests for reactive detection (macro F1) and/or anticipatory prediction (AUC-ROC). The claim is that naturalistic facial video is already sufficient to support generalizable, context-aware, and anticipatory error systems for human-robot interaction.

Core claim

End-to-end models trained on raw, non-anonymized crowdsourced webcam video of spontaneous facial reactions outperform convolutional neural network baselines at both bystander failure detection and anticipatory outcome prediction when evaluated on held-out people who never appeared in training.

What carries the argument

The paired BAD and Bad Idea raw-video datasets, used with a sliding-window protocol that aggregates majority-vote macro F1 for reactive detection and max-probability AUC-ROC for anticipation, forcing models to learn directly from naturalistic pixel streams instead of pre-extracted features.

Load-bearing premise

Facial reactions people make while watching short online videos of failures are informative enough to stand in for the signals a robot would actually see from bystanders during live physical interaction.

What would settle it

Record the same people both while they watch the stimulus videos and while they stand next to a real robot performing the same failures, then test whether models trained only on the video-watching faces still beat chance on the live co-present recordings.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Robots can detect their own mistakes from bystander facial cues without waiting for an explicit user report.
  • Anticipatory models open a path to proactive intervention before a failure fully unfolds.
  • Releasing raw video rather than features lets the community apply modern vision architectures under realistic lighting and camera variation.
  • Subject-independent splits create a reusable public yardstick for comparing future error-detection methods.
  • Cross-dataset transfer can test whether reactive and anticipatory facial signals share transferable structure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If usable anticipatory signal already appears in roughly two-second clips, short-horizon predictive checks could be added to existing robot planners without long observation windows.
  • The strong class imbalance in the reactive dataset means deployed systems will still need calibration or active sampling of rare non-failure cases.
  • Stimulus-watching may understate the multimodal richness of co-present interaction, so pairing these faces with body pose or speech from physical HRI is a natural next test.
  • Modest architectural or training choices already extracted usable signal over CNN baselines, suggesting larger vision models fine-tuned on the same data could raise the performance floor further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper reports the third edition of the ERR@HRI Challenge, which released two complementary crowdsourced webcam datasets (BAD: bystander reactions to observed failures; Bad Idea: anticipatory facial responses before outcomes) as raw, non-anonymized video to support end-to-end multimodal models. It defines three tracks—reactive bystander failure detection, anticipatory outcome prediction, and optional cross-dataset generalization—with subject-independent splits, window-level submission rules, video-level aggregation (majority vote / max positive probability), and primary metrics (macro-F1; AUC-ROC). Convolutional baselines (BadNet; fine-tuned ResNet-34) are provided and shown to be near chance on the held-out tests. Nine teams registered; three submitted valid models that the authors state all surpassed the baselines in at least one track. The paper positions the resources as a durable benchmark for generalizable and anticipatory error detection in HRI.

Significance. If the reported protocol and resources hold as described, the work is a useful incremental contribution to HRI benchmarking: it continues a recurring challenge series, shifts from pre-extracted features to raw video, and adds an anticipatory (pre-failure) task that prior editions lacked. Strengths include subject-independent splits, explicit handling of class imbalance and short clips, released baseline code and hyperparameters, and a commitment to keep materials available. The datasets and evaluation design are the durable value; the factual claim that three teams beat simple CNN baselines is modest but consistent with the near-chance baseline numbers. Ecological validity of stimulus-watching as a proxy for live physical HRI remains an open downstream limitation, not a flaw in the challenge report itself.

major comments (2)
  1. [Section 5; Table 4; Abstract] Section 5 (and the abstract’s results claim) states that three teams submitted valid models that all surpassed the CNN baselines, yet the manuscript provides no quantitative team scores (macro-F1, AUC-ROC, balanced accuracy, temporal metrics), no per-track breakdown, no method summaries, and no Track 3 outcomes. For a challenge paper whose stated purpose includes describing “results,” a leaderboard-style table and brief method notes are load-bearing; without them, “surpassed” is uninterpretable relative to the near-chance baselines in Table 4 (Track 1 macro-F1 0.502; Track 2 AUC 0.564).
  2. [§3.3.3; §3.4; Section 5] Track 3 is defined in §3.3.3 as an optional cross-dataset generalization track “evaluated and awarded separately,” but §3.4–5 give no evaluation protocol (train/test direction, metrics, aggregation) and no results. Either specify the protocol and report outcomes (including null participation) or demote Track 3 to a suggested future analysis so the paper’s claims match what was actually run.
minor comments (6)
  1. [Abstract; §1; §3] Throughout (title, abstract, §1, §3.3) the work is framed as “multimodal,” but the released participant data and baselines are primarily facial video; audio/pose/other streams are not described as provided. Clarify which modalities participants actually received and used.
  2. [§3.1.2; §3.3.2] §3.1.2: Bad Idea labels are participants’ predicted outcomes (well/poorly), not ground-truth scenario outcomes. A one-sentence reminder in the Track 2 task definition and metrics discussion would reduce misreading as failure detection.
  3. [Table 4; §4] Table 4: Report per-class F1 for the baselines in the table (currently only in prose for Track 1) so the imbalance effect is visible at a glance alongside macro-F1.
  4. [Table 3; §3.4; §4] §4: Window settings differ in units and fps (Track 1: 5/2 frames at 5 fps; Track 2: 10/2 at 30 fps). A short note converting both to seconds would aid comparison with the §3.4 window-size cap.
  5. [Abstract; Section 5] §5: “all surpassing our provided baselines in at least one track” should be aligned with the abstract’s stronger “all of which surpassed… baselines” wording once scores are added.
  6. [Abstract; front matter] Minor copy edits: “outcomesbeforefailures” spacing in the abstract; ensure ACM reference page count and DOI placeholders are finalized; consistent capitalization of track names.

Circularity Check

0 steps flagged

No significant circularity; factual challenge report with independent held-out evaluation

full rationale

This is a challenge-description paper, not a theoretical derivation. Its central claims are (i) release of two raw-video datasets with subject-independent train/test splits, (ii) provision of simple CNN baselines (BadNet / fine-tuned ResNet-34) trained under the same protocol, and (iii) the factual report that three teams submitted models that exceeded those baselines on the held-out test sets under fixed metrics (macro-F1 for Track 1, AUC-ROC for Track 2). Self-citations to the source dataset papers and prior ERR@HRI editions are present but merely identify the data provenance; they do not force the numerical results or the ranking. No parameter is fitted and then re-labeled a prediction, no uniqueness theorem is imported from the authors, and no equation reduces to its own inputs by construction. The evaluation protocol (window aggregation, majority vote / max-prob, subject-independent splits) is externally specified and does not circularly encode the outcome. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper is an empirical challenge report, not a first-principles derivation. Load-bearing premises are domain assumptions about social signals and the ecological validity of crowdsourced webcam reactions, plus a handful of free hyperparameters chosen for the baselines. No new physical entities are invented.

free parameters (3)
  • BadNet kernel size / base filters / dropout / LR / epochs (Track 1) = 8 / 64 / 0.7 / 0.0001 / 350
    Chosen via cross-validation on trainval; values (8/64/0.7/1e-4/350) directly determine the reported baseline F1.
  • ResNet-34 fine-tune dropout / LR / epochs / window (Track 2) = 0.7 / 0.001 / 100 / 10 frames
    Hand-selected hyperparameters that set the Track 2 baseline AUC of 0.564.
  • Window size / slide length constraints = ≤2 s (Track 1); majority / max-prob
    Track 1 window capped at 2 s; slide ≤ window; aggregation (majority vote vs max-prob) chosen by organizers and affects ranking.
axioms (3)
  • domain assumption Observable facial/social signals of bystanders or users contain reliable information about robot (or human) errors and can be used for automatic detection.
    Stated in Introduction and Related Work; underpins both tracks and the entire ERR@HRI series.
  • domain assumption Crowdsourced webcam reactions to short stimulus videos of failures (or cut-off actions) are ecologically valid proxies for live HRI error situations.
    Core design choice of both BAD and Bad Idea datasets (Section 3.1); if false, generalization claims weaken.
  • ad hoc to paper Subject-independent train/test splits and the chosen aggregation rules (majority vote for Track 1, max positive probability for Track 2) fairly rank models for real-world generalization.
    Evaluation protocol (Section 3.4); different aggregation would reorder teams.

pith-pipeline@v1.1.0-grok45 · 13449 in / 2693 out tokens · 27382 ms · 2026-07-14T04:41:27.118172+00:00 · methodology

0 comments
read the original abstract

As robots become increasingly integrated into human environments, their ability to detect and respond to errors remains critical for maintaining user trust and interaction quality. While recent advances in machine learning have improved error detection capabilities, most approaches are limited to specific contexts, controlled settings, or pre-extracted features, limiting their generalizability and applicability to real-world conditions. To address this challenge, the third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provided researchers with two complementary datasets that enable end-to-end innovation in methods for both detecting and preventing errors in human-robot interaction. The challenge offered raw, non-anonymized video data from naturalistic settings: (1) the Bystander Affect Detection (BAD) dataset, containing webcam recordings of 45 participants' spontaneous reactions to robot and human failure scenarios; and (2) the Bad Idea dataset, featuring 29 participants' anticipatory facial responses while predicting action outcomes before failures occur. Both datasets were collected via crowdsourcing, capturing the inherent variability of real-world conditions. This naturalistic variability, while challenging, provides an authentic testbed for developing robust error detection systems. Participants developed multimodal machine learning models for bystander reaction detection (Track 1) and anticipatory outcome prediction (Track 2), with an optional cross-dataset generalization track (Track 3). Three teams submitted valid models, all of which surpassed our convolutional neural network baselines. This paper describes the datasets, tasks, baselines, and results of ERR@HRI 3.0, and discusses implications for building generalizable, context-aware, and anticipatory error detection systems for human-robot interaction.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 4 linked inside Pith

  1. [1]

    oh, sorry, i think i interrupted you

    Minja Axelsson, Micol Spitale, and Hatice Gunes. 2024. " oh, sorry, i think i interrupted you": Designing repair strategies for robotic longitudinal well-being coaching. InProceedings of the 2024 ACM/IEEE International Conference on Human- Robot Interaction. 13–22

    2024

  2. [2]

    Alexandra Bremers, Alexandria Pabst, Maria Teresa Parreira, and Wendy Ju. 2024. Using Social Cues to Recognize Task Failures for HRI: Overview, State-of-the-Art, and Future Directions. arXiv:2301.11972 [cs.RO] https://arxiv.org/abs/2301.11972

    Pith/arXiv arXiv 2024

  3. [3]

    Alexandra Bremers, Maria Teresa Parreira, Xuanyu Fang, Natalie Friedman, Adolfo Ramirez-Aristizabal, Alexandria Pabst, Mirjana Spasojevic, Michael Kuni- avsky, and Wendy Ju. 2023. The Bystander Affect Detection (BAD) Dataset for Fail- ure Detection in HRI. 11443-11450 pages. doi:10.1109/IROS55552.2023.10342442

    doi:10.1109/iros55552.2023.10342442 2023

  4. [4]

    Shiye Cao, Jiwon Moon, Amama Mahmood, Victor Nikhil Antony, Ziang Xiao, Anqi Liu, and Chien-Ming Huang. 2025. Interruption handling for conversational robots.Robotics: Science and Systems(2025)

    2025

  5. [5]

    Shiye Cao, Maia Stiber, Amama Mahmood, Maria Teresa Parreira, Wendy Ju, Micol Spitale, Hatice Gunes, and Chien-Ming Huang. 2025. ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations. InProceedings of the 33rd ACM International Conference on Multimedia(Dublin, Ireland)(MM ’25). Association for Computing Machinery,...

    doi:10.1145/3746027.3762073 2025

  6. [6]

    Jennifer Carlson and Robin R Murphy. 2005. How UGVs physically fail in the field.IEEE Transactions on robotics21, 3 (2005), 423–437

    2005

  7. [7]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- geNet: A large-scale hierarchical image database. In2009 IEEE Conference on Com- puter Vision and Pattern Recognition. 248–255. doi:10.1109/CVPR.2009.5206848

    doi:10.1109/cvpr.2009.5206848 2009

  8. [8]

    Shanee Honig and Tal Oron-Gilad. 2018. Understanding and resolving failures in human-robot interaction: Literature review and model development.Frontiers in psychology9 (2018), 861

    2018

  9. [9]

    Dimosthenis Kontogiorgos, Andre Pereira, Boran Sahindal, Sanne van Waveren, and Joakim Gustafson. 2020. Behavioural Responses to Robot Conversational Failures. InProceedings of the 2020 ACM/IEEE International Conference on Human- Robot Interaction(Cambridge, United Kingdom)(HRI ’20). Association for Com- puting Machinery, New York, NY, USA, 53–62. doi:10....

    doi:10.1145/3319502.3374782 2020

  10. [10]

    Dimosthenis Kontogiorgos, Minh Tran, Joakim Gustafson, and Mohammad So- leymani. 2021. A systematic cross-corpus analysis of human reactions to robot conversational failures. InProceedings of the 2021 International Conference on Multimodal Interaction. 112–120

    2021

  11. [11]

    Dong Won Lee, Yubin Kim, Denison Guvenoz, Sooyeon Jeong, Parker Mala- chowsky, Louis-Philippe Morency, Cynthia Breazeal, and Hae Won Park. 2025. ERR@HRI 3.0 ICMI ’26, 5–9 October 2026, Napoli, Italy The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models’ Social Reasoning.arXiv preprint arXiv:2504.13898(2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien- Ming Huang. 2025. User interaction patterns and breakdowns in conversing with llm-powered voice assistants.International Journal of Human-Computer Studies 195 (2025), 103406

    2025

  13. [13]

    Nicole Mirnig, Manuel Giuliani, Gerald Stollnberger, Susanne Stadler, Roland Buchner, and Manfred Tscheligi. 2015. Impact of Robot Actions on Social Signals and Reaction Times in HRI Error Situations. InSocial Robotics, Adriana Tapus, Elisabeth André, Jean-Claude Martin, François Ferland, and Mehdi Ammi (Eds.). Springer International Publishing, Cham, 461–471

    2015

  14. [14]

    Bad Idea, Right?

    Maria Teresa Parreira, Sukruth Gowdru Lingaraju, Adolfo Ramirez-Artistizabal, Alexandra Bremers, Manaswi Saha, Michael Kuniavsky, and Wendy Ju. 2024. “Bad Idea, Right?” Exploring Anticipatory Human Reactions for Outcome Prediction in HRI. In2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN). 2072–2078. doi:10.1109...

    doi:10.1109/ro-man60168.2024.10731310 2024

  15. [15]

    Maha Salem, Gabriella Lakatos, Farshid Amirabdollahian, and Kerstin Dauten- hahn. 2015. Would You Trust a (Faulty) Robot? Effects of Error, Task Type and Personality on Human-Robot Cooperation and Trust. InProceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction(Portland, Oregon, USA)(HRI ’15). Association for Computing...

    doi:10.1145/2696454.2696497 2015

  16. [16]

    Zhonghao Shi, Enyu Zhao, Nathaniel Dennler, Jingzhen Wang, Xinyang Xu, Kaleen Shrestha, Mengxue Fu, Daniel Seita, and Maja Matarić. 2025. HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction. arXiv:2506.20566 [cs.RO] https://arxiv.org/abs/2506. 20566

    Pith/arXiv arXiv 2025

  17. [17]

    Micol Spitale, Minja Axelsson, and Hatice Gunes. 2023. VITA: A Multi-modal LLM-based System for Longitudinal, Autonomous, and Adaptive Robotic Mental Well-being Coaching.arXiv preprint arXiv:2312.09740(2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Micol Spitale, Minja Axelsson, Neval Kara, and Hatice Gunes. 2023. Longitudinal evolution of coachees’ behavioural responses to interaction ruptures in robotic positive psychology coaching. In2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 315–322

    2023

  19. [19]

    Micol Spitale, Maria Teresa Parreira, Maia Stiber, Minja Axelsson, Neval Kara, Garima Kankariya, Chien-Ming Huang, Malte Jung, Wendy Ju, and Hatice Gunes

  20. [20]

    InProceedings of the 26th International Conference on Multimodal Interaction

    ERR@HRI 2024 challenge: Multimodal detection of errors and failures in human-robot interactions. InProceedings of the 26th International Conference on Multimodal Interaction. 652–656

    2024

  21. [21]

    Maia Stiber. 2022. Effective Human-Robot Collaboration via Generalized Robot Error Management Using Natural Human Responses. InProceedings of the 2022 International Conference on Multimodal Interaction(Bengaluru, India)(ICMI ’22). Association for Computing Machinery, New York, NY, USA, 673–678. doi:10. 1145/3536221.3557028

    arXiv 2022

  22. [22]

    Maia Stiber, Russell H Taylor, and Chien-Ming Huang. 2023. On using social signals to enable flexible error-aware hri. InProceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 222–230

    2023

  23. [23]

    Leimin Tian and Sharon Oviatt. 2021. A Taxonomy of Social Errors in Human- Robot Interaction.J. Hum.-Robot Interact.10, 2, Article 13 (feb 2021), 32 pages. doi:10.1145/3439720

    doi:10.1145/3439720 2021