Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Zero-shot vision-language models classify student faces with moderate accuracy; Qwen beats Llama, both miss 'distracted'.

desk verdict A genuinely zero-shot, zero-training baseline for VLM-based academic emotion classification, with honest limitations; the numbers support the descriptive claims, but the sampling and inference layer is thin enough that the Qwen-vs-Llama gap should be treated as provisional. read the letter →

arxiv 2506.10334 v1 pith:IQI76YCZ submitted 2025-06-12 cs.CV cs.AI

classification cs.CVcs.AI
keywords academicemotionfacialexpressionrecognitionvisionlanguagemodelzero-shotpromptingOLSFEDonlinelearningLlama-3.2-11B-Vision-InstructQwen2.5-VL-7B-Instruct
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Academic emotions such as confusion, enjoyment, and fatigue shape how well students learn, but recognizing them automatically usually means collecting and labeling data for each new classroom context. This paper asks whether vision-language models can skip that step: with a single text prompt and no fine-tuning, can they classify students' facial expressions from an online-learning video dataset? On 5,000 images spanning confused, distracted, happy, neutral, and tired faces, it reports moderate overall performance (F1 of 0.45 for Llama-3.2-11B-Vision-Instruct and 0.52 for Qwen2.5-VL-7B-Instruct), with both models strong on happy and weak on distracted. The authors frame the results as preliminary, noting the test covers only a subset of one dataset, only facial expressions, and only two open models.

What carries the argument

The machinery is zero-shot prompting of a vision-language model: a single prompt that combines an instruction to classify the student's expression, explicit definitions of the five target categories, and the input image, with the model forced to answer with one category word only. The task container is the OLSFED dataset, a corpus of 31,115 images of 82 Asian students watching instructional videos, from which the paper samples 1,000 images per expression. No weights are updated; the entire method rests on the model transferring its pretrained visual-linguistic knowledge to an unseen five-way classification.

What would settle it

Re-running the same prompt on the full OLSFED set with per-student stratified splits and a fixed seed, and separately having independent raters re-label a random subset of the 5,000 images, would settle the claim: if Qwen no longer beats Llama, or happy no longer outperforms the other classes, the reported F1 ranking and per-class conclusions would be falsified.

Watch

Extended reading notes

Core claim

The central finding is that a zero-shot prompt can push general-purpose vision-language models to recognize five academic facial expressions, but only at moderate accuracy. Qwen2.5-VL-7B-Instruct reaches an overall F1 of 0.5197 and accuracy of 0.5546, ahead of Llama-3.2-11B-Vision-Instruct's F1 of 0.4519 and accuracy of 0.5118. The per-class pattern is consistent: happy stands out (F1 0.92 for Llama and 0.83 for Qwen), neutral and tired sit in a middling band, and distracted collapses to F1 scores of 0.04 and 0.01 because the models almost never assign that label. Qwen2.5-VL-7B-Instruct also picks out confused expressions with an F1 of 0.65, which the authors connect to the practical goal of identifying lecture content that confuses students.

Load-bearing premise

The results assume the dataset's labels are correct and that the randomly chosen 1,000 images per expression stand in for the full set; the paper gives no evidence that labels are reliable, no detail on how the 82 students are distributed across the sample, and no sampling seed.

Editorial extensions

If this is right

  • A zero-training emotion signal is available for online-learning video: the same prompt can be run on new images without a local labeling campaign, though at roughly 0.5 F1 rather than supervised accuracy.
  • Happy expressions can be trusted as a positive-emotion indicator: both models score above 0.8 F1 on this class, so a simple happy-detector is immediately usable.
  • Distracted behavior should not be measured with these models as prompted: F1 below 0.05 means the zero-shot approach is effectively blind to this class.
  • Qwen2.5-VL-7B-Instruct's confusion F1 of 0.65 gives a plausible low-cost way to flag video moments that puzzle students, if the result repeats on the full dataset.
  • Because the method needs no retraining, it can move across classrooms and camera setups faster than supervised classifiers, but the paper's own limitations paragraph warns that a broader dataset and more models are needed before generalizing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the near-zero recall for distracted suggests the label prior, not visual evidence, is the bottleneck; adding concrete negative examples to the prompt is a cheap next test.
  • Beyond the paper: a per-student split of the 1,000-image sample could change the class-level F1s, since no stratification or seed is reported.
  • Beyond the paper: allowing the models to abstain or to explain their choice would separate prompt-design effects from model capability on confusion and tiredness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a preliminary zero-shot evaluation of two open-source vision-language models, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, on the task of classifying students' academic emotions from facial expressions. Using 5,000 randomly selected images from the OLSFED dataset (1,000 per class across confused, distracted, happy, neutral, and tired), the authors prompt each model with a fixed instruction plus class definitions and report accuracy, precision, recall, and F1. They find that both models achieve moderate overall performance, that Qwen2.5-VL outperforms Llama-3.2, that both models perform best on the happy class, and that both fail to detect the distracted class (near-zero recall), while Qwen2.5-VL performs relatively well on confused expressions. The paper concludes that VLMs offer a promising zero-training baseline for academic emotion recognition, with limitations acknowledged for future work.

Significance. If the reported results are robust, the study provides a useful zero-shot baseline for five-class academic emotion classification on OLSFED and demonstrates that off-the-shelf VLMs can be applied in educational settings without fine-tuning. The evaluation is externally grounded: labels come from the OLSFED dataset, model outputs come from fixed pre-trained checkpoints with a single prompt, and no parameters are fitted to the evaluation data, so circularity is not a concern. The authors are transparent about the exploratory nature of the work. The main value lies in establishing a reference point for future zero-shot or few-shot work, though the lack of statistical rigor currently limits the strength of the comparative and per-class conclusions.

major comments (3)
  1. [§3.1 and §4, Table 1 and Figures 3–5] The central comparative claims—Qwen outperforming Llama and the per-class patterns—rest on point estimates from a single random subsample of 5,000 images with no stated random seed, no stratification by the 82 students, and no confidence intervals or significance tests. Because OLSFED images are video frames from 82 students, the images are not independent; a random subsample may overrepresent particular students or sessions, and the effective sample size for model comparison could be far smaller than 5,000. The observed accuracy difference (0.5546 vs. 0.5118) and F1 difference (0.5197 vs. 0.4519) may be within sampling variability. Please provide the sampling seed, report results with bootstrap confidence intervals or student-level aggregation, and include a confusion matrix with raw counts so that the stability of the ranking and the per-class gaps can be assessed.
  2. [§3.2 and §4, 'distracted' category] The claim that both models 'fail to detect distracted behavior' is based on recall values of 0.02 and 0.01 despite precision of 1.00, meaning the models made very few positive predictions for this class (on the order of 10–20 out of 1,000). The 'distracted' category is defined behaviorally (looking away, turning away) rather than as a facial expression, so the near-zero recall may reflect an ambiguity or mismatch between the prompt's definition and the OLSFED annotation protocol, rather than a pure VLM capability failure. To support the conclusion, please report the full confusion matrix, the number of predicted 'distracted' instances, and, if feasible, perform a small re-annotation or label-reliability check on a subset of the 'distracted' images to establish that the ground-truth labels are consistent.
  3. [§1, §5, and Table 1] The characterization of the models' performance as 'moderate' is uncalibrated without a baseline. A random classifier on the five-way task would achieve 20% accuracy, and a majority-class classifier would also be near that range if classes are balanced; the reported 51.18% and 55.46% accuracies need context to be interpretable. The paper would be substantially strengthened by comparing the zero-shot VLMs with a simple supervised baseline (e.g., a CNN trained on OLSFED) or at least a majority-class baseline, so that the practical value of the zero-shot approach can be judged. Without such a comparison, the conclusion that VLMs 'offer a promising solution' to generalization issues is not fully supported.
minor comments (4)
  1. [Figure captions (Figures 3, 4, 5)] The captions state 'on four facial expressions' but the figures and the paper use five categories (confused, distracted, happy, neutral, tired); please correct the captions to say 'five facial expressions.'
  2. [§3.2] The paper does not specify the exact inference settings used with the two models, such as temperature, maximum number of new tokens, or the model checkpoint dates/commits. Please add these details to ensure the results are reproducible.
  3. [§3.1] The phrase 'randomly selected 1,000 images' should be accompanied by the random seed and a description of the sampling procedure (e.g., whether selection was done per student or from the full collection) to enable reproducibility and to allow assessment of potential student-level bias.
  4. [§4] The text states that both models achieve high precision scores of '1.00 and 1.00' for the distracted class; while the figures corroborate this, reporting actual numbers (e.g., 10/10) rather than only the rounded values would be more informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation uses off-the-shelf zero-shot VLMs against an external dataset, with no fitted parameters and no load-bearing self-citation.

full rationale

The paper's central claims are that two open-source VLMs achieve moderate zero-shot performance on the OLSFED academic facial expression dataset, that Qwen2.5-VL outperforms Llama-3.2, and that both models recognize 'happy' well but fail on 'distracted.' The derivation chain is: take ground-truth labels from OLSFED (an externally published dataset by Bian et al. [3]), randomly subsample 1,000 images per class, run two pre-trained checkpoints with a fixed hand-written prompt, and compute standard metrics. No parameter is fitted to the evaluation data, no model is fine-tuned on the target labels, and no claim is derived from a quantity that depends on the target result. The prompt definitions are simply the five category names with commonsense wording; they are not estimated from the labels and no prompt variant is selected based on the reported metrics. The dataset citation [3] is not authored by the present authors, and the model citations [19, 26] are external model cards. There is no self-citation chain and no uniqueness theorem imported from prior work. The only author-supplied input is the prompt wording, but nothing in the paper suggests that the prompt was tuned on the test subset; even if it were, that would be a methodology weakness rather than circularity because the labels remain external and the models are fixed. Concerns about random sampling without a seed, lack of student-level stratification, and label reliability are validity threats, not circularity: they question whether the reported numbers generalize, but they do not make the metrics equivalent to the inputs. Under the stated criteria, the analysis is self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of the OLSFED ground-truth labels, the representativeness of a random 1,000-image subset per class, and the premise that facial expressions operationalize the five academic emotions. No free parameters or invented entities are introduced; the pre-trained model weights and hand-written prompt definitions are external choices, not fitted parameters.

assumptions (3)
  • domain assumption The OLSFED dataset's ground-truth labels for confused, distracted, happy, tired, and neutral are correct and represent students' academic emotions.
    The paper relies on these labels to compute accuracy, precision, recall, and F1; no label quality check or inter-rater reliability is reported.
  • domain assumption Facial expressions alone are a valid indicator of the five academic emotion categories in an online learning context.
    The study equates academic emotion with facial expression, ignoring other modalities; acknowledged by authors in limitations.
  • domain assumption The random subset of 1,000 images per class is representative of the full dataset and independent of any student-specific bias.
    The selection is described as random but no seed or stratification is given; if images from the same student are overrepresented, metrics may be inflated or distorted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions." pith.science (2026). https://pith.science/paper/IQI76YCZ

@misc{pith2026250610334,
  author       = {Pith},
  title        = {Pith review of: Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQI76YCZ}},
  note         = {Machine review of arXiv:2506.10334}
}
read the original abstract

Students' academic emotions significantly influence their social behavior and learning performance. Traditional approaches to automatically and accurately analyze these emotions have predominantly relied on supervised machine learning algorithms. However, these models often struggle to generalize across different contexts, necessitating repeated cycles of data collection, annotation, and training. The emergence of Vision-Language Models (VLMs) offers a promising alternative, enabling generalization across visual recognition tasks through zero-shot prompting without requiring fine-tuning. This study investigates the potential of VLMs to analyze students' academic emotions via facial expressions in an online learning environment. We employed two VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, to analyze 5,000 images depicting confused, distracted, happy, neutral, and tired expressions using zero-shot prompting. Preliminary results indicate that both models demonstrate moderate performance in academic facial expression recognition, with Qwen2.5-VL-7B-Instruct outperforming Llama-3.2-11B-Vision-Instruct. Notably, both models excel in identifying students' happy emotions but fail to detect distracted behavior. Additionally, Qwen2.5-VL-7B-Instruct exhibits relatively high performance in recognizing students' confused expressions, highlighting its potential for practical applications in identifying content that causes student confusion.

Figures

Figures reproduced from arXiv: 2506.10334 by the authors.

Figure 1
Figure 1. Five facial expressions in the OLSFED dataset[ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The prompt designed for VLMs. of the classification results. The context component provides definitions for the five facial expressions to enhance classi￾fication accuracy. Finally, the input component consists of the student’s images to be analyzed. 4. RESULTS [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. F1 scores of of Llama-3.2-11B-Vision-Instruct and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Precision scores of Llama-3.2-11B-Vision-Instruct [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 37 canonical work pages

  1. [1]

    Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions

    INTRODUCTION Academic emotions refer to the emotions and feelings ex- perienced by students in educational settings, such as en- joyment of learning, pride in success, and test anxiety [23]. These emotions have been found to significantly impact stu- dents’ learning outcomes [2, 9, 25]. For instance, a meta- analysis indicates a significantly positive cor...

  2. [2]

    The Role of Emotions in Stu- dents’ Learning and Achievement,

    RELA TED WORK In 1998, the American Educational Research Association held an annual meeting on “The Role of Emotions in Stu- dents’ Learning and Achievement,” which marked the begin- ning of significant attention being paid to students’ emotions by researchers. Numerous studies have since found that stu- dents experience a variety of emotions in academic ...

  3. [3]

    Specifically, we utilized the Online Learning Spontaneous Facial Expression Database (OLS- FED) [3] as our academic emotion dataset

    METHOD 3.1 Dataset In this study, we analyzed students’ academic emotions through their facial expressions. Specifically, we utilized the Online Learning Spontaneous Facial Expression Database (OLS- FED) [3] as our academic emotion dataset. This dataset captures the facial expressions of 82 students while they watched instructional videos in an online lea...

  4. [4]

    Specifically, Llama-3.2- 11B-Vision-Instruct achieves 0.4519 in F1 score, 0.5118 in accuracy, 0.6473 in precision, and 0.5118 in recall, respec- tively

    RESULTS Table 1 presents the overall performance of Llama-3.2-11B- Vision-Instruct and Qwen2.5-VL-7B-Instruct in classifying five academic facial expressions. Specifically, Llama-3.2- 11B-Vision-Instruct achieves 0.4519 in F1 score, 0.5118 in accuracy, 0.6473 in precision, and 0.5118 in recall, respec- tively. In contrast, Qwen2.5-VL-7B-Instruct achieves ...

  5. [5]

    However, these mod- els face challenges in generalizing well across different con- Table 1: The overall performance of Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct

    DISCUSSION AND CONCLUSION To automatically and accurately analyze students’ academic emotions, researchers have utilized supervised machine learn- ing algorithms to train various models. However, these mod- els face challenges in generalizing well across different con- Table 1: The overall performance of Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Ins...

  6. [6]

    L. Cen, F. Wu, Z. L. Yu, and F. Hu. A real-time speech emotion recognition system and its application in online learning. InEmotions, technology, design, and learning, pages 27–46. Elsevier, 2016

  7. [7]

    S. M. S. A. Abdullah, S. Y. A. Ameen, M. A. Sadeeq, and S. Zeebaree. Multimodal emotion recognition using deep learning.Journal of Applied Science and Technology Trends, 2(01):73–79, 2021

  8. [8]

    Alshareef, I

    N. Alshareef, I. Fletcher, and S. Giga. The role of emotions in academic performance of undergraduate medical students: a narrative review.BMC Medical Education, 24(1):907, 2024

Show all 38 references
  1. [9]

    C. Bian, Y. Zhang, F. Yang, W. Bi, and W. Lu. Spontaneous facial expression database for academic emotion inference in online learning.IET Computer Vision, 13(3):329–337, 2019

  2. [10]

    Bordes, R

    F. Bordes, R. Y. Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma˜ nas, Z. Lin, A. Mahmoud, B. Jayaraman, et al. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024

  3. [11]

    Castellano, L

    G. Castellano, L. Kessous, and G. Caridakis. Emotion recognition through multiple modalities: face, body gesture, speech.Affect and Emotion in Human-Computer Interaction: From Theory to Applications, pages 92–103, 2008

  4. [12]

    L. Guo, W. Wang, T. Wang, W. Li, M. Gong, S. Zhang, W.-H. Zhang, and C. Lu. Association of emotional and behavioral problems with single and multiple suicide attempts among chinese adolescents: modulated by academic performance.Journal of affective disorders, 258:25–32, 2019

  5. [13]

    Im- ages of students’ performance during learning can be used to recognize their academic emotions through facial expres- sions, gestures, and postures [5, 18]

    and computer-supported collaborative learning [31]. Im- ages of students’ performance during learning can be used to recognize their academic emotions through facial expres- sions, gestures, and postures [5, 18]. Furthermore, videos of classroom teaching can be employed to ana...

  6. [14]

    Chiang and C.-J

    W.-W. Chiang and C.-J. Liu. Scale of academic emotion in science education: Development and validation.International Journal of Science Education, 36(6):908–928, 2014

  7. [15]

    I. Dubovi. Cognitive and emotional engagement while learning with vr: The perspective of multimodal methodology.Computers & Education, 183:104495, 2022

  8. [16]

    S. Ekornes. The impact of perceived psychosocial environment and academic emotions on higher education students’ intentions to drop out.Higher Education Research & Development, 41(4):1044–1059, 2022

  9. [17]

    X. Feng, Y. Wei, X. Pan, L. Qiu, and Y. Ma. Academic emotion classification and recognition method for large-scale online learning environment—based on a-cnn and lstm-att deep learning pipeline method.International journal of environmental research and public health, 17(6):1941, 2020

  10. [18]

    Govaerts and J

    S. Govaerts and J. Gr´ egoire. Development and construct validation of an academic emotions scale. International Journal of Testing, 8(1):34–54, 2008

  11. [19]

    Huang, Z

    C. Huang, Z. Han, M. Li, X. Wang, and W. Zhao. Sentiment evolution with interaction levels in blended learning environments: Using learning analytics and epistemic network analysis.Australasian Journal of Educational Technology, 37(2):81–95, 2021

  12. [20]

    Huber and M

    K. Huber and M. Bannert. What happens to your body during learning with computer-based environments? exploring negative academic emotions using psychophysiological measurements.Journal of Computers in Education, 10(1):189–215, 2023

  13. [21]

    E. E. Ketonen, V. Salonen, K. Lonka, and K. Salmela-Aro. Can you feel the excitement? physiological correlates of students’ self-reported emotions.British Journal of Educational Psychology, 93:113–129, 2023

  14. [22]

    Kubsch, D

    M. Kubsch, D. Caballero, and P. Uribe. Once more with feeling: emotions in multimodal learning analytics. InThe multimodal learning analytics handbook, pages 261–285. Springer, 2022

  15. [23]

    Lei and Y

    H. Lei and Y. Cui. Effects of academic emotions on achievement among mainland chinese students: A meta-analysis.Social Behavior and Personality: an international journal, 44(9):1541–1553, 2016

  16. [24]

    J. X.-Y. Lek and J. Teo. Academic emotion classification using fer: A systematic review.Human Behavior and Emerging Technologies, 2023(1):9790005, 2023

  17. [25]

    Llama-3.2-11b-vision-instruct, September 2024

    Meta. Llama-3.2-11b-vision-instruct, September 2024

  18. [26]

    R. Pekrun. Academic emotions. InHandbook of motivation at school, pages 120–144. Routledge, 2016

  19. [27]

    Pekrun and M

    R. Pekrun and M. B ¨uhner. Self-report measures of academic emotions. InInternational handbook of emotions in education, pages 561–579. Routledge, 2014

  20. [28]

    Pekrun, T

    R. Pekrun, T. Goetz, A. C. Frenzel, P. Barchfeld, and R. P. Perry. Measuring emotions in students’ learning and performance: The achievement emotions questionnaire (aeq).Contemporary educational psychology, 36(1):36–48, 2011

  21. [29]

    Pekrun, T

    R. Pekrun, T. Goetz, W. Titz, and R. P. Perry. Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research.Educational psychologist, 37(2):91–105, 2002

  22. [30]

    Rajan, P

    S. Rajan, P. Chenniappan, S. Devaraj, and N. Madian. Facial expression recognition techniques: a comprehensive survey.IET Image Processing, 13(7):1031–1040, 2019

  23. [31]

    J. Tan, J. Mao, Y. Jiang, and M. Gao. The influence of academic emotions on learning effects: A systematic review.International journal of environmental research and public health, 18(18):9678, 2021

  24. [32]

    Q. Team. Qwen2.5-vl, January 2025

  25. [33]

    Wei and J

    X. Wei and J. Yin. The application of physiological feedback to the evaluation of academic emotion: A literature review. In2021 Tenth International Conference of Educational Innovation through Technology (EITT), pages 346–349. IEEE, 2021

  26. [34]

    Xiang, Z

    A. Xiang, Z. Qi, H. Wang, Q. Yang, and D. Ma. A multimodal fusion network for student emotion recognition based on transformer and tensor product. In2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pages 1–4. IEEE, 2024

  27. [35]

    H. Zeng, X. Shu, Y. Wang, Y. Wang, L. Zhang, T.-C. Pong, and H. Qu. Emotioncues: Emotion-oriented visual summarization of classroom videos.IEEE transactions on visualization and computer graphics, 27(7):3168–3181, 2020

  28. [36]

    Zhang, J

    J. Zhang, J. Huang, S. Jin, and S. Lu. Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

  29. [37]

    Zheng, M

    L. Zheng, M. Long, J. Niu, and L. Zhong. An automated group learning engagement analysis and feedback approach to promoting collaborative knowledge building, group performance, and socially shared regulation in cscl.International Journal of Computer-Supported Collaborative Lea...

  30. [38]

    X. Zhu, C. Guo, H. Feng, Y. Huang, Y. Feng, X. Wang, and R. Wang. A review of key technologies for emotion analysis using multimodal information. Cognitive Computation, 16(4):1504–1530, 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.