User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals

Jun Baba; Junya Nakanishi; Ryo Miyoshi; Takuya Iwamoto; Yuki Okafuji

REVIEW 4 major objections 5 minor 35 references

User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a Transformer-based model can estimate a user's experience with a robot from facial expressions and voice, and that it outperforms third-party human evaluators in this task.

desk verdict Useful new UX dataset and a sensible modeling framework, but the headline claim of beating human evaluators is unestablished because the humans got far less training data than the model. read the letter →

arxiv 2507.23544 v1 pith:2YS4IXKL submitted 2025-07-31 cs.RO cs.CVcs.HC

Ryo Miyoshi , Yuki Okafuji , Takuya Iwamoto , Junya Nakanishi , Jun Baba This is my paper

classification cs.ROcs.CVcs.HC

keywords userexperienceestimationhuman-robotinteractionmulti-instancelearningmultimodalsocialsignalsfacialexpressionrecognitionvoiceanalysisTransformerUXevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that user experience (UX) in human-robot interaction can be estimated automatically from a user's facial expressions and voice, and that this estimate can be more accurate than the judgment of a third-party human observer. The authors create a dataset of 22 participants interacting with two robots, collect 15-item UX questionnaires after each of 220 interactions, and train a Transformer-based model that processes video clips and audio spectrograms with a two-level multi-instance learning structure. The reported results show the multimodal model reaching about 33% accuracy on a seven-level scale and 70% on a three-level scale, compared with about 23% and 64% for human evaluators. If these results hold, robots could monitor and react to users' experience in real time instead of relying on post-interaction surveys.

What carries the argument

The core mechanism is a two-stage multi-instance learning architecture built from shared CNNs and Transformer encoders. The user's video is split into short clips; each clip's frames pass through a CNN and a first Transformer to produce a clip-level vector, and a second Transformer aggregates these vectors to capture long-term interaction patterns. The audio track is converted to a mel-spectrogram, divided into patches, and encoded by a CNN plus Transformer in the same patchwise fashion. The resulting global audio and vision representations are concatenated and fed to an MLP that predicts one of seven UX scores, and a CLS token in each Transformer is what aggregates the sequence into a single representation. Attention rollout is applied to inspect which clips and audio patches the model relies on, showing that expressive episodes and tonal changes receive high attention.

What would settle it

Conduct the same comparison with human evaluators trained on videos from the same twenty participants used to train the model, then measure Acc.7 and Acc.3 on held-out participants; if human accuracy then equals or exceeds the model's, the reported advantage is an artifact of unequal training exposure.

Watch

Extended reading notes

Core claim

The central discovery is that a two-stage multi-instance Transformer, fed with cropped face images and mel-spectrogram patches of the user's voice, can predict self-reported UX questionnaire responses more accurately than third-party human evaluators. On the seven-level scale the proposed model averages about 33% accuracy versus about 23% for human raters, and on the three-level scale it averages about 70% versus 64%, while both substantially exceed chance. The paper also finds that the multimodal model beats each unimodal variant on most questionnaire items, and that attention is spread over multiple video clips, concentrating on moments of expressive change rather than uniformly across the interaction. The authors interpret this as evidence that UX, being a multi-faceted construct, is better estimated by modeling temporal dynamics of social signals than by momentary observations.

Load-bearing premise

The finding that the model outperforms human evaluators depends on the assumption that human raters who trained on only two participants' videos are a fair human benchmark; if a person given the same amount of training data as the model were equally or more accurate, the central claim would no longer hold.

Editorial extensions

If this is right

Robots could estimate a user's experience continuously during an interaction and adapt their behavior in real time, reducing reliance on post-interaction questionnaires.
A single camera and microphone provide sufficient sensing for the multimodal UX estimator, making deployment feasible for service robots.
The same system can predict scores for all 15 questionnaire items spanning attractiveness, stimulation, interactivity, and satisfaction, covering a broader construct than engagement or sentiment alone.
The multi-instance design shows that preserving temporal structure matters: momentary snapshot-based predictors would be weaker, so interaction-aware architectures are preferable for UX estimation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not pursued in the paper is to use the predicted UX as a reward signal for reinforcement learning, letting a robot learn behaviors that optimize estimated user satisfaction without human labels.
The human baseline was given far less training data than the model, so a symmetric human experiment would likely narrow the gap; this is a limitation to address before claiming general superiority over human judgment.
The attention findings suggest a testable hypothesis: UX-relevant moments are localized in time, so an estimator that crops interactions to the most expressive segments might match full-video accuracy at a much lower computational cost.
If the approach replicates across different robot platforms, interaction types, and user demographics, it would support a broader claim that self-reported experience leaves measurable traces in brief expressive behavior, worth testing in multi-party or long-duration conversations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this one. First, it ships a genuinely new resource: 220 human-robot interactions with per-interaction 15-item UX questionnaires, designed to elicit positive and negative experiences. That is a real contribution for the HRI community. Second, the central claim in the abstract and conclusion — that the model 'outperforms third-party human evaluators' — does not survive contact with the experimental protocol. The human evaluators each watched only two participants' videos as training; the model was trained on 12 or 22 participants depending on the comparison. That is not a fair race, and the paper gives no significance tests or confidence intervals for either side. The gap between 33% and 23% Acc.7 could easily shrink or vanish if the humans saw comparable labeled data.

The modeling work is competent. A two-stage Transformer with clip-level and video-level aggregation is a standard video-classification stack, and calling it multi-instance learning is a bit generous, but it is a reasonable architecture for the task. The ablation shows the multimodal model edges out unimodal (35.1 vs 34.5 Acc.7), which is honest but tiny, and the paper does not test whether that difference or the differences from chance are statistically meaningful. The attention analysis is interesting but anecdotal.

The soft spots are mostly fixable. The human-baseline protocol is the load-bearing one; it needs to be matched (humans get the same number of training videos) or the claim needs to be downgraded to 'comparable to third-party evaluators.' The participant-count inconsistency (22 participants, then '22 were male, and one was female') is a sloppy typo that undermines confidence in the reporting. And for a paper whose main asset is a dataset, not releasing the data or code is a serious limitation.

Who should read this? Researchers working on automated UX or affect estimation in HRI. The dataset, if released, would be useful even if the current model results are modest. Does it deserve a serious referee? Yes. The contribution is real, the flaws are fixable, and a thorough review could push the authors toward a much stronger paper. I would not reject it on sight; I would send it out, but with a clear expectation that the human comparison be redone or reined in.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Transformer-based multi-instance learning model that estimates seven-point questionnaire UX ratings from facial-expression videos and voice in human-robot interaction. The authors introduce a new dataset of 220 interactions from 22 participants with two robots, evaluate unimodal and multimodal variants under subject-independent 4-fold cross-validation, and compare the best model against seven third-party human evaluators. They report that the model outperforms the human evaluators and provide attention analyses to interpret the model's focus.

Significance. If the reported results were robust, the paper would make a useful contribution: a new HRI UX dataset, a sensible weakly supervised temporal architecture, and an explicit human-baseline comparison are all valuable for affective HRI. The subject-independent evaluation protocol and the attention analysis are strengths. However, the central claim, that the method outperforms third-party human evaluators, depends on a severely unmatched baseline and is not supported by the reported statistics; the multimodal advantage is also numerically tiny and untested. With those load-bearing issues unresolved, the contribution is not yet established.

major comments (4)

[V-B, Figs. 7-8] The central claim that the model 'outperforms third-party human evaluators' is not supported because the human and model baselines are not matched in training data. The text states that 'each evaluator watched videos of two participants as training data,' while the model was trained under subject-independent 4-fold cross-validation on the full dataset; for the comparison, the paper reports only that the model's performance was averaged over 12 participants and does not specify how many participants' labels the model saw. This conflates model quality with a substantial training-data advantage and makes the reported 10-point Acc.7 gap uninterpretable. The comparison also lacks significance tests or confidence intervals for the model, and the human error bars show only the best and worst evaluator rather than the uncertainty of the mean.
[III] The participant counts are internally inconsistent. The text says 'The experiment was conducted with 22 participants' and then 'Among the participants, 22 were male, and one was female,' which cannot both be true. Section V-B then states that 'This experiment used data from 12 of the 22 participants,' but the main dataset description reports 220 data points (22 participants x 10 scenarios). The paper must clarify the actual total sample size, the gender breakdown, and which analyses use the full 22 participants versus the 12-participant subset; this directly affects the validity of the reported cross-validation and human-comparison results.
[V-A, Figs. 5-6] The claimed benefit of multimodality is not statistically demonstrated. The average Acc.7 values are 34.5% for the vision-only model, 33.6% for the audio-only model, and 35.1% for the multimodal model, a difference of roughly 0.6 percentage points. No error bars, confidence intervals, or significance tests are reported for any model, and the statement that 'each model performed significantly better than chance' is made without a statistical test. Similarly, the Acc.3 differences among models are small and could easily be noise. The authors should report per-fold variability or permutation tests before claiming that the multimodal approach is effective.
[V-B] The human-evaluation reporting is insufficient for a fair baseline comparison. The paper states that error bars in the third-party evaluation graph represent the maximum and minimum accuracy values, i.e., the best and worst evaluator, which is not a measure of the uncertainty of the mean human performance. No inter-rater reliability statistic (e.g., Cohen's kappa or Krippendorff's alpha) is reported, so it is unclear whether the human ratings are reliable enough to serve as a meaningful benchmark. This matters because the headline claim rests entirely on this comparison.

minor comments (5)

[IV, Eq. (4)] The notation in Eq. (4) is self-referential: it defines Fvi = Transformer([Fvi, CLSi]), but the intended input is the frame-level feature vectors Ci, so it should be Fvi = Transformer([Ci, CLSi]).
[IV, Section on Vision Encoder] There is a grammatical and typographical issue: 'The feature vectors Ci extracted from individual frames are processed by the Transformer encoder The feature vectors Ci are processed along with a CLS token...' The sentence is broken and should be rewritten.
[Fig. 3] The questionnaire in Fig. 3 contains typos: 'want to use agein' should be 'want to use again', and the label 'conversation difficultconversation easy' is missing a separator.
[V-A, Experimental Settings] The key hyperparameters (16 video clips, 16 frames per clip, 16 patch images) are presented without any sensitivity analysis or justification, leaving open how dependent the results are on these choices.
[V-A] The statement that the proposed model outperforms the other models on most questionnaire items is not fully supported by Fig. 6, where the multimodal model is not the best on every item; the text should acknowledge the item-level variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the UX-estimation claim is an empirical supervised-learning result, with the human-comparison asymmetry being a validity concern rather than a circular step.

full rationale

Walked the paper's derivation chain: the UX dataset (Sec. III) is independently collected from self-reported questionnaires; the proposed model (Sec. IV) is a standard Transformer/MIL supervised architecture trained with cross-entropy loss (Eq. 7); Acc.7 and Acc.3 are conventional accuracy metrics computed on held-out participants via subject-independent 4-fold cross-validation. No fitted parameter is renamed as a prediction: the model's output is the questionnaire score itself, and the input features (facial images, mel-spectrograms) are not defined in terms of the target UX labels. The comparison with third-party human evaluators (Sec. V-B) is empirical and, while it raises a real protocol-fairness concern because evaluators watched only two participants' training videos whereas the model was trained on substantially more data, that is an experimental-validity limitation rather than a circularity. Self-citations such as [3], [4], and [22] are used only to describe prior robot behaviors and context, and none is load-bearing for the UX-estimation performance claim. The central claim therefore does not reduce to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper rests on standard supervised learning assumptions plus domain-specific operationalizations. No free parameters in the mathematical sense; hyperparameters and evaluation thresholds are chosen by hand.

free parameters (3)

Number of video clips and frames per clip = 16 clips, 16 frames each
Chosen in Section V experimental settings; not justified by ablation.
Acc.3 quantization thresholds = 1-2 negative, 3-5 neutral, 6-7 positive
Defined in Section V; compresses 7-point scale into 3 classes and affects all reported Acc.3 numbers.
Questionnaire item subset = 15 items from UEQ
Section III-B selects only Attractiveness and Stimulation plus custom items; this operationalization defines the UX target.

assumptions (4)

domain assumption Self-reported answers to the selected 15 items constitute a valid measure of UX in HRI.
Section III-B adapts UEQ; no validation against other UX instruments.
domain assumption Facial expression and voice contain sufficient information to predict UX; posture, gaze, and head movement are negligible.
Section IV states posture/gaze/head showed minimal variation, so those modalities were dropped.
domain assumption The 10 scripted robot behaviors elicit the intended positive/negative UX.
Section III-A lists behaviors; no manipulation check reported.
domain assumption Subject-independent 4-fold cross-validation with 22 participants yields stable accuracy estimates.
Small sample; no confidence intervals or significance tests reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals." pith.science (2026). https://pith.science/paper/2YS4IXKL

@misc{pith2026250723544,
  author       = {Pith},
  title        = {Pith review of: User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YS4IXKL}},
  note         = {Machine review of arXiv:2507.23544}
}

read the original abstract

In recent years, the demand for social robots has grown, requiring them to adapt their behaviors based on users' states. Accurately assessing user experience (UX) in human-robot interaction (HRI) is crucial for achieving this adaptability. UX is a multi-faceted measure encompassing aspects such as sentiment and engagement, yet existing methods often focus on these individually. This study proposes a UX estimation method for HRI by leveraging multimodal social signals. We construct a UX dataset and develop a Transformer-based model that utilizes facial expressions and voice for estimation. Unlike conventional models that rely on momentary observations, our approach captures both short- and long-term interaction patterns using a multi-instance learning framework. This enables the model to capture temporal dynamics in UX, providing a more holistic representation. Experimental results demonstrate that our method outperforms third-party human evaluators in UX estimation.

Figures

Figures reproduced from arXiv: 2507.23544 by the authors.

**Figure 2.** Example of participant images captured by the camera [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Overview of the proposed UX estimation model [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (6 more)

**Figure 6.** Figure 6: As shown in Fig. 5, the average Acc.7 across all [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 5.** Figure 5: Acc.7 of each model Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Ave. Red：Attractiveness，Yellow：Stimulation，Purple：Interactiveness，Green：Satisfaction 80.5% 76.4% 70.0% 68.6% 72.3% 75.5% 70.0% 71.4% 74.6% 68.6% 60.5% 68.2% 66.8% 71.4% 70.9% 71.0% 75.5% 77.3% 70.5%…

**Figure 6.** Figure 6: Acc.3 of each model questionnaire items was 34.5% for the vision-only model, 33.6% for the audio-only model, and 35.1% for the multimodal model using both facial images and voice. Since the chance rate for Acc.7 was 14.3%, each model performed significantly better tha…

**Figure 7.** Figure 7: Acc.7 of each model and a third-party model [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Acc.3 of each model and a third-party estimation [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: Examples of attention obtained from each model [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

[1]

Home-assistant robot for an aging society,

K. Yamazaki, R. Ueda, S. Nozawa, M. Kojima, K. Okada, K. Mat- sumoto, M. Ishikawa, I. Shimoyama, and M. Inaba, “Home-assistant robot for an aging society,” Proceedings of the IEEE , vol. 100, no. 8, pp. 2429–2441, 2012

work page 2012
[2]

Robots serve humans in pub- lic places—kejia robot as a shopping assistant,

Y . Chen, F. Wu, W. Shuai, and X. Chen, “Robots serve humans in pub- lic places—kejia robot as a shopping assistant,” International Journal of Advanced Robotic Systems , vol. 14, no. 3, p. 1729881417703569, 2017

work page 2017
[3]

Teleoperated robot acting autonomous for better customer satisfaction,

J. Baba, S. Sichao, J. Nakanishi, I. Kuramoto, K. Ogawa, Y . Yoshikawa, and H. Ishiguro, “Teleoperated robot acting autonomous for better customer satisfaction,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems , 2020, pp. 1–8

work page 2020
[4]

Playful recommendation: Sales promotion that robots stimulate pleasant feelings instead of product explanation,

T. Iwamoto, J. Baba, J. Nakanishi, K. Hyodo, Y . Yoshikawa, and H. Ishiguro, “Playful recommendation: Sales promotion that robots stimulate pleasant feelings instead of product explanation,” IEEE Robotics and Automation Letters , vol. 7, no. 4, pp. 11 815–11 822, 2022

work page 2022
[5]

Service robots in a bakery shop: A field study,

S. Song, B. Jun, J. Nakanishi, Y . Yoshikawa, and H. Ishiguro, “Service robots in a bakery shop: A field study,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2022, pp. 134–140

work page 2022
[6]

Recognizing users feedback from non-verbal communicative acts in conversational recommender systems,

B. De Carolis, M. de Gemmis, P. Lops, and G. Palestra, “Recognizing users feedback from non-verbal communicative acts in conversational recommender systems,” Pattern Recognition Letters, vol. 99, pp. 87– 95, 2017

work page 2017
[7]

Nonverbal communication in humans,

F. B. Mandal, “Nonverbal communication in humans,” Journal of human behavior in the social environment, vol. 24, no. 4, pp. 417–421, 2014

work page 2014
[8]

Social signal processing: Survey of an emerging domain,

A. Vinciarelli, M. Pantic, and H. Bourlard, “Social signal processing: Survey of an emerging domain,” Image and vision computing, vol. 27, no. 12, pp. 1743–1759, 2009

work page 2009

Show all 35 references

[9]

Engagement in human-agent interaction: An overview,

C. Oertel, G. Castellano, M. Chetouani, J. Nasir, M. Obaid, C. Pelachaud, and C. Peters, “Engagement in human-agent interaction: An overview,” Frontiers in Robotics and AI , vol. 7, p. 92, 2020

2020
[10]

Eval- uating the engagement with social robots,

S. M. Anzalone, S. Boucenna, S. Ivaldi, and M. Chetouani, “Eval- uating the engagement with social robots,” International Journal of Social Robotics, vol. 7, pp. 465–478, 2015

2015
[11]

Personalized estimation of engagement from videos using active learning with deep reinforcement learning,

O. (Oggi) Rudovic, H. Won Park, J. Busche, B. Schuller, C. Breazeal, and R. W. Picard, “Personalized estimation of engagement from videos using active learning with deep reinforcement learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

2019
[12]

Are you still with me? continuous engagement assessment from a robot’s point of view,

F. Del Duchetto, P. Baxter, and M. Hanheide, “Are you still with me? continuous engagement assessment from a robot’s point of view,” Frontiers in Robotics and AI , vol. 7, p. 116, 2020

2020
[13]

Learning on the job: Long-term behavioural adaptation in human-robot interactions,

F. Del Duchetto and M. Hanheide, “Learning on the job: Long-term behavioural adaptation in human-robot interactions,” IEEE Robotics and Automation Letters , vol. 7, no. 3, pp. 6934–6941, 2022

2022
[14]

A survey of multimodal sentiment analysis,

M. Soleymani, D. Garcia, B. Jou, B. Schuller, S.-F. Chang, and M. Pantic, “A survey of multimodal sentiment analysis,” Image and Vision Computing, vol. 65, pp. 3–14, 2017

2017
[15]

Sentiment analysis: from opinion mining to human-agent interaction,

C. Clavel and Z. Callejas, “Sentiment analysis: from opinion mining to human-agent interaction,” IEEE Transactions on affective computing , vol. 7, no. 1, pp. 74–93, 2015

2015
[16]

Assessment of users’ interests in multi- modal dialog based on exchange unit,

S. Tomimasu and M. Araki, “Assessment of users’ interests in multi- modal dialog based on exchange unit,” in Proceedings of the Workshop on Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction, 2016, pp. 33–37

2016
[17]

Multimodal learning for identifying opportunities for em- pathetic responses,

L. Tavabi, K. Stefanov, S. Nasihati Gilani, D. Traum, and M. So- leymani, “Multimodal learning for identifying opportunities for em- pathetic responses,” in 2019 International Conference on Multimodal Interaction, 2019, pp. 95–104

2019
[18]

Morse: Multimodal sentiment analysis for real-life settings,

Y . Yao, V . P ´erez-Rosas, M. Abouelenien, and M. Burzo, “Morse: Multimodal sentiment analysis for real-life settings,” in Proceedings of the 2020 International Conference on Multimodal Interaction , 2020, pp. 387–396

2020
[19]

Effects of physiological signals in different types of multimodal sentiment estimation,

S. Katada, S. Okada, and K. Komatani, “Effects of physiological signals in different types of multimodal sentiment estimation,” IEEE Transactions on Affective Computing , 2022

2022
[20]

P. S. P. Rex Hartson, The UX Book: Agile UX Design for a Quality User Experience. Morgan Kaufmann, Cambridge, 2018

2018
[21]

User experience questionnaire handbook,

M. Schrepp, “User experience questionnaire handbook,” All you need to know to apply the UEQ successfully in your project , 2015

2015
[22]

Behavioral assessment of a humanoid robot when at- tracting pedestrians in a mall,

Y . Okafuji, Y . Ozaki, J. Baba, J. Nakanishi, K. Ogawa, Y . Yoshikawa, and H. Ishiguro, “Behavioral assessment of a humanoid robot when at- tracting pedestrians in a mall,” International journal of social robotics, vol. 14, no. 7, pp. 1731–1747, 2022

2022
[23]

User experience in social robots,

E. S. Shourmasti, R. Colomo-Palacios, H. Holone, and S. Demi, “User experience in social robots,” Sensors, vol. 21, no. 15, p. 5052, 2021

2021
[24]

Does the personality of consumers influence the assessment of the experience of interaction with social robots?

S. Forgas-Coll, R. Huertas-Garcia, A. Andriella, and G. Aleny `a, “Does the personality of consumers influence the assessment of the experience of interaction with social robots?” International Journal of Social Robotics, pp. 1–21, 2022

2022
[25]

Speech-based recognition of self-reported and observed emotion in a dimensional space,

K. P. Truong, D. A. Van Leeuwen, and F. M. De Jong, “Speech-based recognition of self-reported and observed emotion in a dimensional space,” Speech communication, vol. 54, no. 9, pp. 1049–1063, 2012

2012
[26]

Fser: Deep convolutional neural networks for speech emotion recognition,

B. F. P. Dossou and Y . K. S. Gbenou, “Fser: Deep convolutional neural networks for speech emotion recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, October 2021, pp. 3533–3538

2021
[27]

AST: Audio Spectrogram Transformer,

Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in Proc. Interspeech 2021 , 2021, pp. 571–575

2021
[28]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

2009
[29]

Ex- pression snippet transformer for robust video-based facial expression recognition,

Y . Liu, W. Wang, C. Feng, H. Zhang, Z. Chen, and Y . Zhan, “Ex- pression snippet transformer for robust video-based facial expression recognition,” Pattern Recognition, vol. 138, p. 109368, 2023

2023
[30]

Former-dfer: Dynamic facial expression recog- nition transformer,

Z. Zhao and Q. Liu, “Former-dfer: Dynamic facial expression recog- nition transformer,” in Proceedings of the 29th ACM International Conference on Multimedia , 2021, pp. 1553–1561

2021
[31]

Logo-former: Local-global spatio-temporal transformer for dynamic facial expression recognition,

F. Ma, B. Sun, and S. Li, “Logo-former: Local-global spatio-temporal transformer for dynamic facial expression recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

2023
[32]

Training deep networks for facial expression recognition with crowd-sourced label distribution,

E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” in Proceedings of the 18th ACM international conference on multimodal interaction , 2016, pp. 279–283

2016
[33]

Openface 2.0: Facial behavior analysis toolkit,

T. Baltrusaitis, A. Zadeh, Y . C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, 2018, pp. 59–66

2018
[34]

Human vs. computer in scene and object recognition,

A. Borji and L. Itti, “Human vs. computer in scene and object recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 113–120

2014
[35]

Quantifying attention flow in transform- ers,

S. Abnar and W. Zuidema, “Quantifying attention flow in transform- ers,” arXiv preprint arXiv:2005.00928 , 2020

2005 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Home-assistant robot for an aging society,

K. Yamazaki, R. Ueda, S. Nozawa, M. Kojima, K. Okada, K. Mat- sumoto, M. Ishikawa, I. Shimoyama, and M. Inaba, “Home-assistant robot for an aging society,” Proceedings of the IEEE , vol. 100, no. 8, pp. 2429–2441, 2012

work page 2012

[2] [2]

Robots serve humans in pub- lic places—kejia robot as a shopping assistant,

Y . Chen, F. Wu, W. Shuai, and X. Chen, “Robots serve humans in pub- lic places—kejia robot as a shopping assistant,” International Journal of Advanced Robotic Systems , vol. 14, no. 3, p. 1729881417703569, 2017

work page 2017

[3] [3]

Teleoperated robot acting autonomous for better customer satisfaction,

J. Baba, S. Sichao, J. Nakanishi, I. Kuramoto, K. Ogawa, Y . Yoshikawa, and H. Ishiguro, “Teleoperated robot acting autonomous for better customer satisfaction,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems , 2020, pp. 1–8

work page 2020

[4] [4]

Playful recommendation: Sales promotion that robots stimulate pleasant feelings instead of product explanation,

T. Iwamoto, J. Baba, J. Nakanishi, K. Hyodo, Y . Yoshikawa, and H. Ishiguro, “Playful recommendation: Sales promotion that robots stimulate pleasant feelings instead of product explanation,” IEEE Robotics and Automation Letters , vol. 7, no. 4, pp. 11 815–11 822, 2022

work page 2022

[5] [5]

Service robots in a bakery shop: A field study,

S. Song, B. Jun, J. Nakanishi, Y . Yoshikawa, and H. Ishiguro, “Service robots in a bakery shop: A field study,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2022, pp. 134–140

work page 2022

[6] [6]

Recognizing users feedback from non-verbal communicative acts in conversational recommender systems,

B. De Carolis, M. de Gemmis, P. Lops, and G. Palestra, “Recognizing users feedback from non-verbal communicative acts in conversational recommender systems,” Pattern Recognition Letters, vol. 99, pp. 87– 95, 2017

work page 2017

[7] [7]

Nonverbal communication in humans,

F. B. Mandal, “Nonverbal communication in humans,” Journal of human behavior in the social environment, vol. 24, no. 4, pp. 417–421, 2014

work page 2014

[8] [8]

Social signal processing: Survey of an emerging domain,

A. Vinciarelli, M. Pantic, and H. Bourlard, “Social signal processing: Survey of an emerging domain,” Image and vision computing, vol. 27, no. 12, pp. 1743–1759, 2009

work page 2009