Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Multimodal Sentiment Analysis based on Video and Audio Inputs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining audio and video models improves emotion recognition.

desk verdict Honest and transparent, but the leaky RAVDESS split and test-set-peeking make the fusion claim unsupported. read the letter →

arxiv 2412.09317 v1 pith:JTJLBNYZ submitted 2024-12-12 cs.SD cs.AIcs.CVcs.MMeess.AS

Antonio Fernandez , Suzan Awinat This is my paper · ORCID
classification cs.SDcs.AIcs.CVcs.MMeess.AS
keywords sentimentanalysisemotionrecognitionmultimodalfusionaudioclassificationvideowav2vec2ViTdecision-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that emotion recognition from videos can be built by fusing the outputs of two separately fine-tuned unimodal transformer models, one for audio and one for video, rather than training a joint multimodal network. Using the CREMA-D audio dataset and the RAVDESS video dataset, the authors fine-tune wav2vec2-large and ViViT, then combine their per-emotion probability outputs with five simple decision rules. On a 105-video test sample, the combined systems match or exceed the better unimodal model, with averaging and a rule-based logic performing best when the two models have similar accuracy. The paper concludes that the combined approach gives acceptable results on the emotion recognition tasks, while flagging that consistency needs further testing.

What carries the argument

The mechanism is decision-level fusion of per-emotion probability vectors. Each unimodal model emits a probability distribution over the six emotions (anger, disgust, fear, happiness, neutral, and sadness), and the fusion rules operate purely on these vectors and on each model's accuracy or prediction confidence: the mean probability, a weighted mean scaled by model accuracy, a threshold that trusts the video model when its top probability exceeds 0.7, a dynamic weighting based on inverse total confidence, and a rule that returns the agreed emotion when both models are confident or otherwise the higher-confidence prediction. No fusion parameters are learned.

What would settle it

Train the video and fusion models on a RAVDESS split where no actor appears in both training and test (for example, hold out six actors entirely), then rerun the five fusion rules; if the combined accuracy falls to or below the better unimodal model's accuracy, the claimed fusion benefit is refuted.

Watch

Extended reading notes

Core claim

The central claim is that decision-level fusion of two independently fine-tuned unimodal transformers yields acceptable emotion recognition accuracy. The audio model (wav2vec2-large) and the video model (ViViT) each output a probability distribution over six emotions; the five fusion rules — simple averaging, accuracy-weighted averaging, a video-confidence threshold, dynamic confidence-based weighting, and rule-based logic — convert these two distributions into a single prediction. In the first model version, where both modalities had similar accuracy, averaging and rule-based logic produced the best results; in the second version, where the video model was much stronger, the accuracy-weighted average kept the combined score at or above the video model's level. The paper presents this as evidence that combining video and audio inputs is a usable approach for emotion recognition.

Load-bearing premise

The load-bearing premise is that the RAVDESS train/test split is clean, so scores reflect emotion recognition instead of memory of the same 24 actors and two sentences showing up in both training and test.

Editorial extensions

If this is right

  • If averaging two unimodal probability vectors improves accuracy, multimodal sentiment analysis can be improved without training a fusion network.
  • The rule-based logic, which trusts agreement or the higher-confidence model, performs as well as averaging, so confidence signals carry useful information for combining models.
  • When one modality is much more accurate, the accuracy-weighted average prevents the weaker model from pulling the combined result down.
  • The six-emotion, single-culture, controlled-recording setup means the approach still needs to be tested on diverse, in-the-wild data before it can be generalized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The RAVDESS test set is drawn from the same 24 actors and two sentences used in training, so the reported accuracy, especially the video model's roughly 88% in the second version, may partly reflect identity and sentence memorization rather than emotion generalization; a person-disjoint split would test this.
  • A direct experiment would be to hold out all videos from a subset of actors and rerun the five fusion rules; if the fusion advantage disappears, the central claim would be weakened.
  • Applying the same framework to out-of-distribution videos — different lighting, backgrounds, languages, or cultures — would reveal whether the acceptable results persist beyond the controlled RAVDESS setting.
  • The authors' own discussion of the EU AI Act suggests that any deployment, such as the proposed therapy robot, would need regulatory review before use.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multimodal emotion recognition pipeline that combines a fine-tuned wav2vec2-large audio model and a fine-tuned ViViT video model, and it compares five decision-level fusion strategies (simple averaging, weighted averaging, confidence-level thresholding, dynamic confidence-based weighting, and rule-based logic) on a test sample of 105 RAVDESS video files. The authors report that fusion improves over the unimodal baselines, that averaging and rule-based logic are the best fusion methods, and that the combined video+audio approach yields "acceptable results" on emotion recognition. The central claim is that simple decision-level fusion of two independently fine-tuned unimodal transformers can achieve usable emotion recognition without training a fusion network.

Significance. If the claims were established, the paper would provide a lightweight alternative to learned multimodal fusion, which is a meaningful practical contribution. The authors provide links to Kaggle notebooks for training and evaluation, which is helpful for reproducibility. However, the experimental design has two load-bearing weaknesses: the RAVDESS test set is likely leaked in terms of actor identity and sentence content, and the fusion methods are selected after observing results on the same test set. These issues call the central conclusion into question. The paper also contains no statistical analysis, and the audio and video models are evaluated under different domain-shift conditions. The significance of the claimed finding is therefore contingent on a reworked evaluation that the current manuscript does not provide.

major comments (4)
  1. [Section 3.3] The construction of the 105-file test set introduces actor and sentence leakage that can inflate video accuracy. RAVDESS contains only 24 actors, each reciting the same two statements across all emotions, intensities, and repetitions. Removing only the video-only counterparts of the 105 test files leaves in training many videos from the same actors and the same two statements, frequently with the same emotion at a different intensity or repetition. The ViViT video model can therefore exploit actor identity, lighting, and sentence-specific mouth movements to reach the reported ~88% accuracy without learning emotion-general visual cues. Because all fusion results are computed on this same test set, the central claim that combining video and audio inputs improves emotion recognition is not established. A leave-actors-out split, or at least a split that ensures no actor and no statement appear in both training and test, is required to support the claim.
  2. [Section 3.3] The four additional fusion strategies (weighted average, confidence threshold, dynamic weighting, rule-based logic) were created after inspecting the averaging method's results on the same 105-file test set. The abstract and Section 3.3 state, "After disparity in the results... another test framework is created," and Section 3.3 explicitly says both V1 and V2 were run on the same 105-file RAVDESS sample. This is a post hoc selection of fusion rules on the test data, so the comparison among the five methods is not a fair out-of-sample evaluation. The conclusion that averaging and rule-based logic are the best methods may simply reflect overfitting to these 105 files. The paper should pre-register the fusion rules or use a separate validation set for method selection and a different held-out test set for final evaluation.
  3. [Sections 3.1, 3.2] The audio and video unimodal models are evaluated under different domain-shift conditions, which confounds the comparison that motivates the fusion design. The audio model is fine-tuned on CREMA-D and tested on RAVDESS audio, whereas the video model is fine-tuned on RAVDESS and tested on RAVDESS video. The lower audio accuracy (around 59-72%) relative to video accuracy (~88%) may reflect a train/test domain shift rather than an inherent difference in modality difficulty. Since the weighted average, confidence threshold, and dynamic weighting methods are all motivated by the video model's higher accuracy, the validity of this motivation depends on an apples-to-apples comparison. The paper should either fine-tune the audio model on RAVDESS audio as well or explicitly analyze and control for the domain shift.
  4. [Section 3.3, Discussion] All conclusions about fusion performance are based on a single test set of 105 files, with no confidence intervals, error bars, or statistical tests. The differences among the five fusion methods are small (a few percentage points), and the ranking of averaging and rule-based logic as the best methods could easily be within sampling noise. The paper should report per-class accuracy, confusion matrices, and bootstrapped confidence intervals or a significance test (e.g., McNemar's test) to support the claimed ranking. Without this, the qualitative conclusions in the Discussion are not strongly supported.
minor comments (5)
  1. [Abstract] There are typographical errors, including "avarage" for "average" and "the models that been used" for "the models that were used." The abstract also says five methods are used but lists only four; the averaging method is presumably the fifth, but this should be explicit.
  2. [Section 2.2] The dataset name is inconsistently written as "RA VDESS" in several places; it should be "RAVDESS" throughout.
  3. [Section 2.3] There are typos in the text: "prepossessed" should be "preprocessed," and "tdqm" should be "tqdm." Also, the library name "A V" should be "PyAV" or "av".
  4. [Figures 2-6] The figures show accuracy bars but the numeric values are not reported in the captions or axes; the reader cannot determine the exact accuracy differences between methods. Please add a table with exact values or label the bars.
  5. [Throughout] The paper uses "sentiment analysis" and "emotion recognition" interchangeably. The task is emotion recognition (six discrete emotion classes), not sentiment analysis (typically positive/negative). Clarifying this terminology would avoid confusion with the sentiment-analysis literature cited in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation is present; the paper is an empirical study, and the test-set selection concern is a validity issue rather than circular reasoning.

full rationale

The paper contains no mathematical derivation chain in which an output is shown to equal an input by construction. The unimodal models are fine-tuned independently on external datasets (CREMA-D for audio, RAVDESS for video), and their outputs are combined by simple decision-level fusion rules. The fusion rules are not fitted parameters that are then renamed as predictions; they are hand-specified aggregation procedures (averaging, weighted averaging, thresholding, dynamic weighting, rule-based logic). The authors do state that after seeing a disparity in V2 results they created additional test frameworks, which is a form of test-set peeking and weakens the generalizability claim, but this is an experimental design flaw, not circular reasoning. The self-citations to Kaggle notebooks are references to the authors' own training and testing code, but the paper also reports the numerical results in its tables and text; these citations do not smuggle in an unverified premise that is then used to justify the same premise. No quoted equation or construction shows that the conclusion was assumed in the inputs. Under the hard rule that circularity must be exhibited as a specific reduction, no circular step can be identified, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new entities are proposed. The free parameters are fusion thresholds and test-set accuracies used as weights; each is fitted to or selected on the same test data. The axioms above are the domain assumptions needed for the reported accuracies to mean what the paper claims.

free parameters (3)
  • confidence threshold on video model = 0.7
    Chosen for the Confidence Level Threshold method in Section 3.3; not justified by a validation set.
  • agreement confidence threshold in rule-based logic = 0.5
    Used in Section 3.3 to decide whether both models agree enough to return the agreed emotion; chosen by hand.
  • model accuracy weights in weighted average = V1 audio ~72.59%, video similar; V2 video ~88%, audio ~59%
    The Weighted Average method scales probabilities by each model's test accuracy; these accuracies come from the same or related test evaluations, so the method's parameters are fitted to the test data.
assumptions (4)
  • domain assumption Softmax probabilities from wav2vec2 and ViViT are comparable and can be averaged or thresholded across modalities.
    The whole fusion framework in Section 3.3 treats the two models' probability vectors as commensurable confidence scores without calibration.
  • domain assumption The six emotion labels are equivalent across CREMA-D and RAVDESS after dropping calm and surprise.
    Section 2.3 removes calm and surprised from RAVDESS to match CREMA-D labels; the paper does not analyze whether the remaining categories are distributed or expressed similarly.
  • domain assumption File-level random split of RAVDESS gives an unbiased test.
    Section 2.2 randomly samples 105 files and removes only video-only counterparts; this assumes no actor or statement overlap between train and test, which is false given 24 actors and repeated statements.
  • domain assumption Test accuracy on a 105-sample subset is a stable estimate.
    Section 3.3 draws conclusions from 105 files without confidence intervals or repeated sampling.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Sentiment Analysis based on Video and Audio Inputs." pith.science (2026). https://pith.science/paper/JTJLBNYZ

@misc{pith2026241209317,
  author       = {Pith},
  title        = {Pith review of: Multimodal Sentiment Analysis based on Video and Audio Inputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTJLBNYZ}},
  note         = {Machine review of arXiv:2412.09317}
}
read the original abstract

Despite the abundance of current researches working on the sentiment analysis from videos and audios, finding the best model that gives the highest accuracy rate is still considered a challenge for researchers in this field. The main objective of this paper is to prove the usability of emotion recognition models that take video and audio inputs. The datasets used to train the models are the CREMA-D dataset for audio and the RAVDESS dataset for video. The fine-tuned models that been used are: Facebook/wav2vec2-large for audio and the Google/vivit-b-16x2-kinetics400 for video. The avarage of the probabilities for each emotion generated by the two previous models is utilized in the decision making framework. After disparity in the results, if one of the models gets much higher accuracy, another test framework is created. The methods used are the Weighted Average method, the Confidence Level Threshold method, the Dynamic Weighting Based on Confidence method, and the Rule-Based Logic method. This limited approach gives encouraging results that make future research into these methods viable.

Figures

Figures reproduced from arXiv: 2412.09317 by the authors.

Figure 1
Figure 1. The entire process of our module [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. Accuracy of Different Models Using Confidence Level Threshold on the Video Model Method [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Accuracy of Different Models Using Dynamic Weighting Based on Confidence Method [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 21 canonical work pages

  1. [1]

    Tang, Z., Cho, J., Nie, Y ., & Bansal, M. (2022). TVLT: Textless vision-language transformer. Advances in Neural Information Processing Systems, 35, 9617-9632

    work page 2022

  2. [2]

    Al-Azani, S., & El-Alfy, E. S. M. (2020). Enhanced video analytics for sentiment analysis based on fusing textual, auditory and visual infor- mation. IEEE Access, 8, 136843-136857

    work page 2020

  3. [3]

    Pereira, M., P ´adua, F., Pereira, A., Benevenuto, F., & Dalip, D. (2016). Fusing audio, textual, and visual features for sentiment analysis of news videos. In Proceedings of the International AAAI Conference on Web and Social Media (V ol. 10, No. 1, pp. 659-662)

    work page 2016

  4. [4]

    A., Yousef, A

    Abdu, S. A., Yousef, A. H., & Salem, A. (2021). Multimodal video sentiment analysis using deep learning approaches, a survey. Information Fusion, 76, 204-226

    work page 2021

  5. [5]

    Wang, D., Guo, X., Tian, Y ., Liu, J., He, L., & Luo, X. (2023). TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis. Pattern Recognition, 136, 109259

    work page 2023

  6. [6]

    (2021, October)

    Yuan, Z., Li, W., Xu, H., & Yu, W. (2021, October). Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 4400-4407)

    work page 2021

  7. [7]

    Cao H, Cooper DG, Keutmann MK, Gur RC, Nenkova A, Verma R. (2014). CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset. IEEE Trans Affect Comput. 5(4):377-390. doi: 10.1109/TAFFC.2014.2336244. PMID: 25653738; PMCID: PMC4313618

    work page arXiv 2014

  8. [8]

    R., & Russo, F

    Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RA VDESS) [Data set]. En PLoS ONE (1.0.0, V ol. 13, N´umero 5, p. e0196391). Zenodo. https://doi.org/10.5281/zenodo.1188976

    doi:10.5281/zenodo.1188976 2018

Show all 26 references
  1. [9]

    Baevski, A., Zhou, Y ., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460

    2020

  2. [10]

    Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lu ˇci´c, M., & Schmid, C. (2021). Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6836-6846)

    2021

  3. [11]

    & Chintala, S

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., ... & Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  4. [12]

    P., McVicar, M., Battenberg, E., & Nieto, O

    McFee, B., Ra ffel, C., Liang, D., Ellis, D. P., McVicar, M., Battenberg, E., & Nieto, O. (2015, July). librosa: Audio and music signal analysis in python. In SciPy (pp. 18-24)

    2015

  5. [13]

    & Rush, A

    Wolf, T., Debut, L., Sanh, V ., Chaumond, J., Delangue, C., Moi, A., ... & Rush, A. (2020). Transformers: State-of-the-art natural language processing. In Q. Liu & D. Schlangen (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Syst...

    2020 doi

  6. [14]

    V ., Jernite, Y ., Thakur, A., von Platen, P., Patil, S.,

    Lhoest, Q., del Moral, A. V ., Jernite, Y ., Thakur, A., von Platen, P., Patil, S., ... & Tunstall, L. (2021). Datasets: a community library for natural language processing. arXiv. arXiv preprint arXiv:2109.02846

    2021 arXiv

  7. [15]

    & Duchesnay, ´E

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V ., Thirion, B., Grisel, O., ... & Duchesnay, ´E. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830

    2011

  8. [16]

    & Zheng, X

    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., ... & Zheng, X. (2016). TensorFlow: a system for Large-Scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) (pp. 265-283)

    2016

  9. [17]

    T., Shambrook, B., Van der Wel, C.,

    Boers, M., Lain ´e, J., Reid, M., Fauske, V . T., Shambrook, B., Van der Wel, C., ... & Joy, D. (2017). PyA V Documentation. Available at: https://pyav.org/docs/stable/index.html

    2017

  10. [18]

    K., Altendorf, K., Mary, H., Sheridan, R., Korobov, M.,

    Costa-Luis, C., Larroque, S. K., Altendorf, K., Mary, H., Sheridan, R., Korobov, M., ... & Plavin, A. (2024). tqdm: A fast, Extensible Progress Bar for Python and CLI (v4.66.2). Zenodo. https://doi.org/10.5281/zenodo.3551211

    2024 doi

  11. [19]

    Zulko, E. (2019). MoviePy: Video editing with Python. Retrieved from https: //zulko.github.io/moviepy/

    2019

  12. [20]

    Hwang, J., Hira, M., Chen, C., Zhang, X., Ni, Z., Sun, G., Ma, P., Huang, R., Pratap, V ., Zhang, Y ., Kumar, A., Yu, C.-Y ., Zhu, C., Liu, C., Kahn, J., Ravanelli, M., Sun, P., Watanabe, S., Shi, Y ., Tao, Y ., Scheibler, R., Cornell, S., Kim, S., & Petridis, S. (2023). Torch...

    2023 arXiv

  13. [21]

    Z., Lian, J., Mahadeokar, J., Hwang, J., Chen, J., Goldsborough, P., Roy, P., Narenthiran, S., Watanabe, S., Chintala, S., Quenneville-B´elair, V ., & Shi, Y

    Yang, Y .-Y ., Hira, M., Ni, Z., Chourdia, A., Astafurov, A., Chen, C., Yeh, C.-F., Puhrsch, C., Pollack, D., Genzel, D., Greenberg, D., Yang, E. Z., Lian, J., Mahadeokar, J., Hwang, J., Chen, J., Goldsborough, P., Roy, P., Narenthiran, S., Watanabe, S., Chintala, S., Quennevi...

    2021 arXiv

  14. [22]

    European Parliament. (2024). Artificial Intelligence Act European Parliament legislative resolution of 13 March 2024 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on Artificial Intelligence (Artifi- cial Intellig...

    2024

  15. [23]

    (2nd of May, 2024)

    Fernandez, A. (2nd of May, 2024). Audio Sentiment Analysis Model Training. Available at: https://www.kaggle.com/antoniobfernandez/audio- sentiment-analysis-model-training

    2024

  16. [24]

    (2nd of May, 2024)

    Fernandez, A. (2nd of May, 2024). Video Sentiment Analysis Model Training. Available at: https://www.kaggle.com/code/antoniobfernandez/video-sentiment-analysis-model-training/notebook

    2024

  17. [25]

    (2nd of May, 2024)

    Fernandez, A. (2nd of May, 2024). Multimodal Sentiment Analysis Test Framework V1. Available at: https://www.kaggle.com/code/antoniobfernandez/multimodal-sentiment-analysis-test-framework-v1/notebook

    2024

  18. [26]

    (2nd of May, 2024)

    Fernandez, A. (2nd of May, 2024). Multimodal Sentiment Analysis Test Framework V2. Available at: https://www.kaggle.com/code/antoniobfernandez/multimodal-sentiment-analysis-test-framework-v2/notebook

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.