REVIEW 4 major objections 6 minor 38 references
StandUp4AI: A New Multilingual Dataset for Humor Detection in Stand-up Comedy Videos
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A 334-hour, seven-language stand-up comedy dataset is presented as the largest of its kind, and word-level laughter labeling with an ASR-timestamp fix is shown to beat raw-label baselines.
desk verdict Useful multilingual dataset with a good framing; the ASR enhancement is the weak link and the 'largest' claim needs substantiation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the complementary timestamp-error pattern of the two automatic speech recognition systems used for transcription. Whisper tends to attach a laughter interval to the following word while WhisperX attaches it to the previous one; by taking the intersection of abnormally long word timestamps in the two transcripts, the pipeline recovers a segment that corresponds to the missed laugh, trims it from both neighboring words, and passes the candidate to a random forest trained on acoustic features such as duration, pitch, spectral characteristics, and MFCCs. This machinery simultaneously repairs word timestamps and converts speech-recognition anomalies into additional laughter labels, which is what lets the sequence-labeling model be trained on roughly 128,000 word-level laughter tags.
What would settle it
On a random sample of laughs from each of the seven languages, compare the word-boundary shift assigned by the two speech recognizers; if any language consistently shows both recognizers merging laughter into the same neighboring word rather than opposite ones, the intersection method would miss those laughs and the enhanced-label training signal would be corrupted.
Extended reading notes
Core claim
The paper's central claim is that StandUp4AI is the largest and most linguistically diverse multilingual dataset of live comedy performances, and that it supports a better way to model humor. Instead of classifying a whole utterance as humorous or not, the authors label each word with whether audience laughter begins or continues after it, turning humor detection into continuous word-level sequence labeling that lets a system anticipate laughter as it happens. The empirical discovery underneath this is an annotation fix: around a laugh, one automatic speech recognizer tends to merge the laughter duration into the following word while the other merges it into the previous word, so the intersection of the two long word intervals isolates the laugh. That intersection yields new laughter candidates, a random forest filters them by acoustic features, and the resulting enhanced labels improve both the underlying laughter detector (F1 0.51 to 0.58 on a manually annotated test set) and the downstream humor model (token-level F1 42.4 with a multilingual text model).
Load-bearing premise
The pipeline's extra laughter labels rest on the assumption that the two speech recognizers always attach a laugh's duration to opposite neighboring words, a directional pattern the paper illustrates with only one example and does not verify per language.
Editorial extensions
If this is right
- The 70-video manually annotated test set provides a fixed multilingual benchmark, so future humor models in English, French, Spanish, Italian, Portuguese, Hungarian, and Czech can be compared on identical ground-truth labels.
- A word-level laughter tag gives a conversational agent a signal for when to laugh and when to keep laughing, instead of only telling it that a whole sentence was funny.
- The ASR timestamp-intersection trick is not tied to any one language, so it can be applied to other laughter-labeled speech collections, in principle turning transcription errors into training data.
- Releasing facial Action Units, body poses, and camera-angle features alongside the text and laughter labels opens the stated next step of training multimodal humor models that use delivery and gesture, not just wording.
- The reported gap between automatic and manually annotated test labels means even weak-label training transfers to real laughter, but stronger supervision on the manual set is the direct path to higher performance.
Reading between the lines
- The paper leaves implicit that its word-level laughter tags measure audience response, not humor itself; a joke that gets laughter in one culture or language may be silent in another, so the benchmark may reward models that predict audience behavior rather than comedic content.
- The same complementary-ASR-error logic could likely be extended to other non-speech acoustic events such as applause, gasps, or music, because the mechanism is about voice-activity-detection timestamp distortion rather than laughter specifically; this is a direct experimental extension of the paper's method.
- A more skeptical interpretation of the 42.4 F1 is that the model may be learning prosodic or discourse cues such as sentence boundaries and pauses that correlate with laughter, without representing humor itself; an ablation that controls for punctuation and pause features would test this.
- The dataset's design of many videos per language from a small set of comedy brands means the seven-language comparison may partly reflect channel-specific recording conditions and audience culture; models trained on it should be tested on held-out comedians and venues to see if the multilingual advantage survives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces StandUp4AI, a multilingual dataset of 3,617 stand-up comedy videos (334 hours, 7 languages) with automatic laughter annotations, a manually annotated 70-video test set, and word-level sequence labeling baselines using XLM-RoBERTa. The authors also propose an ASR-based method that combines Whisper and WhisperX outputs to correct timestamp errors around laughter and recover laughter segments missed by an off-the-shelf detector. The best multilingual text-only model reaches 42.4 token-level F1 on the manually annotated test set using the enhanced labels, versus 42.2 with raw labels.
Significance. If the annotation quality and baseline results are confirmed, StandUp4AI would be a valuable resource: it is substantially larger and more diverse than existing stand-up humor corpora, it reframes humor detection as sequence labeling rather than binary classification, and it introduces a reusable ASR-error-based laughter enhancement technique. The public release of code, data, and extracted multimodal features supports reproducibility. However, the paper's quantitative claims are currently too weakly supported to serve as a reference: the headline improvement of the enhancement is 0.2 F1 on average, with negative differences in several languages, and no significance testing or inter-annotator agreement is provided.
major comments (4)
- [§2.4, Table 6] The complementary timestamp-error pattern between Whisper and WhisperX is the central mechanism of the proposed enhancement, yet it is illustrated with only a single schematic example in Table 6. Table 7 provides only aggregate IoU F1 scores (0.51 vs 0.58) for the whole test set, with no per-language results, confidence intervals, or significance tests. Because the dataset is explicitly multilingual and Section 5 concedes that dialectal Spanish can challenge ASR, the paper must show that the error pattern holds per language; otherwise the 'enhanced' laughter labels, which are both a released annotation layer and the training signal for the best baseline, are not established as reliable.
- [§3.2, Table 2] Table 2 does not support the claim that models trained on the ASR-enhanced labels outperform those trained on raw labels. The average improvement is +0.2 F1 (42.2 to 42.4), and the per-language differences are negative for CS (-0.3), EN (-0.1), and ES (-1.0), and zero for IT. No variance estimates, significance tests, or repeated-seed results are reported, so the headline improvement is statistically indistinguishable from noise. Additionally, the monolingual Enhanced models are below the multilingual Raw model for five of seven languages, which weakens the conclusion that the corpus diversity is the driver of the gains.
- [§2.6] The manually annotated test set is the only gold standard used to validate both the laughter detection method and the sequence labeling baselines, but no inter-annotator agreement is reported. The text states that annotations were made 'with precise timestamps at 0.1 seconds' using Audacity, but it does not state how many annotators participated or whether boundary agreement was measured. For a resource paper whose main deliverable is annotated data, the reliability of the gold labels must be quantified; otherwise the absolute F1 values in Tables 2, 3, and 7 cannot be interpreted.
- [Appendix B] The Random Forest candidate filter is trained on 376 manually annotated candidates drawn from 50 videos, but the paper does not report the language distribution of those candidates. If the candidates are concentrated in a subset of the seven languages, the filter's acoustic decision boundary may not transfer to the other languages, and the 'enhanced' annotations would be biased. Please provide per-language candidate counts and per-language filter performance, or explain why the filter is expected to be language-agnostic.
minor comments (6)
- [Abstract] The sentence 'we propose a method for e propose a method' contains a duplicated fragment and should be corrected to 'we propose a method'.
- [§3.2] The phrase 'indicating a second time the quality of ti' appears to be a typo; it should likely read 'the quality of the data'.
- [Appendix C] The phrase 'the principal of the algorithm' should be 'the principle of the algorithm'.
- [Table 2] The column header 'Laughters' is unclear; clarify whether it counts laughter-annotated words or laughter segments, and consider renaming it.
- [§2.5] The label definition 'before the end of next word' is ambiguous in the main text; Appendix A clarifies it, but the main text should give a self-contained definition.
- [References] Some reference entries are incomplete or inconsistently formatted (e.g., 'Mohammed, and Hoque' in the Hasan et al. entry); please normalize the bibliography.
Circularity Check
No significant circularity: the laughter-enhancement pipeline and the sequence-labeling model are each evaluated against a separately manually annotated test set.
full rationale
The paper's derivation chain is: collect videos; run off-the-shelf laughter detection; obtain Whisper/WhisperX transcripts; engineer an ASR-timestamp-error rule to add laughter candidates; train a random forest on 376 manually labelled candidates from 50 videos; insert accepted candidates into the training labels; and finally train sequence-labeling models. The only learned component in the annotation pipeline, the random forest, is trained on manual labels from 50 videos and then evaluated end-to-end on a separate manually annotated 70-video test set (Section 3.1, Table 7: F1 0.58 vs 0.51 for the off-the-shelf model). The sequence-labeling models are likewise evaluated against this same manual test set, not against the auto-annotations used for training; the 42.4 token F1 in Table 2 is thus an independent measurement rather than a fitted re-statement of the training labels. The ASR error-pattern assumption in Section 2.4 is an empirical hypothesis, not a conclusion imported from a self-citation; the paper validates it on the held-out test set, and the Limitations section flags per-language ASR robustness as a limitation rather than asserting it as a theorem. No self-citations are load-bearing, and no equation or label is defined in terms of the quantity it is used to predict. The small candidate-filter training set and the lack of per-language breakdown are statistical robustness concerns, not circularity.
Assumptions & free parameters
free parameters (4)
- Random Forest hyperparameters =
n_estimators=50, max_depth=13, min_samples_split=2
- Duration threshold for candidate verification =
0.5 seconds
- IoU threshold for laughter detection evaluation =
0.2
- Sequence cutting stripe =
128 words
assumptions (5)
- domain assumption Omine et al. (2024) off-the-shelf laughter detection model performs adequately on all 7 languages and diverse recording conditions.
- domain assumption Whisper and WhisperX timestamp errors around laughter are systematic and complementary: Whisper merges laughter duration with the next word, WhisperX with the previous.
- domain assumption Audience laughter is a valid and sufficient proxy for humor in stand-up comedy.
- domain assumption Manual annotations on the 70-video test set are reliable.
- domain assumption The acoustic features used for the Random Forest generalize across languages and recording conditions.
Cite this review
Pith. "Pith review of StandUp4AI: A New Multilingual Dataset for Humor Detection in Stand-up Comedy Videos." pith.science (2026). https://pith.science/paper/OKMFVSZI
@misc{pith2026250518903,
author = {Pith},
title = {Pith review of: StandUp4AI: A New Multilingual Dataset for Humor Detection in Stand-up Comedy Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/OKMFVSZI}},
note = {Machine review of arXiv:2505.18903}
}
read the original abstract
Aiming towards improving current computational models of humor detection, we propose a new multimodal dataset of stand-up comedies, in seven languages: English, French, Spanish, Italian, Portuguese, Hungarian and Czech. Our dataset of more than 330 hours, is at the time of writing the biggest available for this type of task, and the most diverse. The whole dataset is automatically annotated in laughter (from the audience), and the subpart left for model validation is manually annotated. Contrary to contemporary approaches, we do not frame the task of humor detection as a binary sequence classification, but as word-level sequence labeling, in order to take into account all the context of the sequence and to capture the continuous joke tagging mechanism typically occurring in natural conversations. As par with unimodal baselines results, we propose a method for e propose a method to enhance the automatic laughter detection based on Audio Speech Recognition errors. Our code and data are available online: https://tinyurl.com/EMNLPHumourStandUpPublic
Figures
Reference graph
Works this paper leans on
-
[1]
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for...
work page 2016
-
[2]
The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition
Shahin Amiriparian, Lukas Christ, Alexander Kathan, Maurice Gerczuk, Niklas M \" u ller, Steffen Klug, Lukas Stappen, Andreas K \" o nig, Erik Cambria, Björn Schuller, and Simone Eulitz. 2024. http://arxiv.org/abs/2406.07753 The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition
work page Pith review arXiv 2024
-
[3]
Shahin Amiriparian, Lukas Christ, Andreas K \" o nig, Alan Cowen, Eva Maria Me ner, Erik Cambria, and Björn W. Schuller. 2023. https://doi.org/10.1145/3581783.3610943 MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects . MM 2023 - Proceedings of the 31st ACM International Conferen...
-
[4]
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. https://doi.org/10.21437/Interspeech.2023-78 WhisperX: Time-Accurate Speech Transcription of Long-Form Audio . Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2023-Augus:4489--4493
-
[5]
Dario Bertero and Pascale Fung. 2016. Deep Learning of Audio and Language Features for Humor Prediction . Lrec, pages 496--501
work page 2016
-
[6]
B rian M c F ee, C olin R affel, D awen L iang, D aniel P . W . E llis, M att M c V icar, E ric B attenberg, and O riol N ieto. 2015. https://doi.org/10.25080/Majora-7b98e3ed-003 librosa: A udio and M usic S ignal A nalysis in P ython . In P roceedings of the 14th P ython in S cience C onference , pages 18 -- 24
-
[7]
Brandon Castellano. 2014. https://github.com/Breakthrough/PySceneDetect Pyscenedetect
work page 2014
-
[8]
Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mohammad Soleymani. 2024. https://doi.org/10.1109/WACV57701.2024.00802 LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis . Proceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024, pages 8190--8200
arXiv 2024
Show all 38 references
-
[9]
Dushyant Singh Chauhan, Gopendra Vikram Singh, Navonil Majumder, Amir Zadeh, Asif Ekbal, Pushpak Bhattacharyya, Louis Philippe Morency, and Soujanya Poria. 2021. https://doi.org/10.1145/3462244.3479959 M2H2: A Multimodal Multiparty Hindi Dataset for Humor Recognition in Conver...
2021
-
[10]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang,...
2019 arXiv
-
[11]
Lei Chen and Chong Min Lee. 2017. https://doi.org/10.18653/v1/w17-5009 Predicting audience's laughter during presentations using convolutional neural network . EMNLP 2017 - 12th Workshop on Innovative Use of NLP for Building Educational Applications, BEA 2017 - Proceedings of ...
2017 doi
-
[12]
Akshat Choube and Mohammad Soleymani. 2020. https://doi.org/10.1145/3382507.3418891 Punchline Detection using Context-Aware Hierarchical Multimodal Fusion . ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction, pages 675--679
2020
-
[13]
u ller, Andreas K \
Lukas Christ, Shahin Amiriparian, Alexander Kathan, Niklas M \" u ller, Andreas K \" o nig, and Björn W. Schuller. 2022. http://arxiv.org/abs/2209.14272 Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results . XX(X):1--18
2022 arXiv
-
[14]
MMPose Contributors. 2020. Openmmlab pose estimation toolbox and benchmark. https://github.com/open-mmlab/mmpose
2020
-
[15]
Jon Gillick, Wesley Deng, Kimiko Ryokai, and David Bamman. 2021. https://doi.org/10.21437/Interspeech.2021-353 Robust Laughter Detection in Noisy Environments . In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 1...
2021 doi
-
[16]
Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, Mohammed , and Hoque . 2019. http://arxiv.org/abs/1904.06618 UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
2019 arXiv
-
[17]
Lee Hyun, Kim Sung-Bin, Seungju Han, Youngjae Yu, and Tae Hyun Oh. 2024. SMILE: Multimodal Dataset for Understanding Laughter with Language Models . Findings of the Association for Computational Linguistics: NAACL 2024 - Findings, pages 1149--1167
2024
-
[18]
Antonios Kalloniatis and Panagiotis Adamidis. 2024. Computational humor recognition: a systematic literature review. Artificial Intelligence Review, 58(2):43
2024
-
[19]
Diederik Kingma and Jimmy Ba. 2014. https://doi.org/http://doi.acm.org.ezproxy.lib.ucf.edu/10.1145/1830483.1830503 Adam: A Method for Stochastic Optimization . International Conference on Learning Representations, pages 1--13
2014
-
[20]
Anna Kuznetsova and Carlo Strapparava. 2024. Multimodal and Multilingual Laughter Detection in Stand-Up Comedy Videos . 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings, pages 11...
2024
-
[21]
Guillaume Lample and Alexis Conneau. 2019. http://arxiv.org/abs/1901.07291 Cross-lingual Language Model Pretraining
2019 arXiv
-
[22]
Zhi Song Liu, Robin Courant, and Vicky Kalogeiton. 2024. https://doi.org/10.1007/s11263-024-02000-2 FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild . International Journal of Computer Vision, 132(8):2885--2906
2024 doi
-
[23]
Bogdan Ludusan and Barbara Schuppler. 2022. https://doi.org/10.18653/v1/2022.sigdial-1.8 To laugh or not to laugh? The use of laughter to mark discourse structure . SIGDIAL 2022 - 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, Proceedings of the C...
2022 doi
-
[24]
Rod A Martin and Thomas Ford. 2018. The psychology of humor: An integrative approach. Academic press
2018
-
[25]
Hiroki Nakayama. 2018. https://github.com/chakki-works/seqeval \ seqeval \ : A Python framework for sequence labeling evaluation
2018
-
[26]
Taisei Omine, Kenta Akita, and Reiji Tsuruno. 2024. https://doi.org/10.21437/interspeech.2024-1644 Robust Laughter Segmentation with Automatic Diverse Data Synthesis . In Interspeech, September, pages 4748--4752
2024 doi
-
[27]
Patro, Mayank Lunayach, Deepankar Srivastava, Sarvesh Sarvesh, Hunar Singh, and Vinay P
Badri N. Patro, Mayank Lunayach, Deepankar Srivastava, Sarvesh Sarvesh, Hunar Singh, and Vinay P. Namboodiri. 2021. https://doi.org/10.1109/WACV48630.2021.00062 Multimodal humor dataset: Predicting laughter tracks for sitcoms . Proceedings - 2021 IEEE Winter Conference on Appl...
2021
-
[28]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesna...
2012 doi
-
[29]
Amruta Purandare and Diane Litman. 2006. Humor : Prosody Analysis and Automatic Recognition for F * R * I * E * N * D * S * . In EMNLP, July, pages 208--215
2006
-
[30]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision . Proceedings of Machine Learning Research, 202:28492--28518
2023
-
[31]
Victor Raskin. 1979. Semantic mechanisms of humor. In Annual Meeting of the Berkeley Linguistics Society, pages 325--335
1979
-
[32]
Beatrice Turano and Carlo Strapparava. 2022. Making People Laugh like a Pro: Analysing Humor Through Stand-Up Comedy . 2022 Language Resources and Evaluation Conference, LREC 2022, (June):5206--5211
2022
-
[33]
Caleb Warren and A Peter McGraw. 2016. Differentiating what is humorous from what is not. Journal of Personality and Social Psychology, 110(3):407
2016
-
[34]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. http://arxiv.org/abs/1910.03771 HuggingFace's Transformers: State-of-the-art Natural Language Processing
2019 arXiv
-
[35]
Robert S Wyer and James E Collins. 1992. A theory of humor elicitation. Psychological review, 99(4):663
1992
-
[36]
Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, and Louis-philippe Morency. 2020. CMU-MOSEAS : A Multimodal Language Dataset for Spanish , Portuguese , German and French . In EMNLP, volume 1, pages 1801--1812
2020
-
[37]
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-philippe Morency. 2019. Social-IQ : A Question Answering Benchmark for Artificial Social Intelligence . In CVPR, pages 8807--8817
2019
-
[38]
funny how?
Nima Zargham, Vino Avanesi, Leon Reicherts, Ava Elizabeth Scott, Yvonne Rogers, and Rainer Malaka. 2023. “funny how?” a serious look at humor in conversational agents. In Proceedings of the 5th International Conference on Conversational User Interfaces, pages 1--7
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.