Pith. sign in

REVIEW 4 major objections 4 minor 47 references

Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that model reliability and how the model is framed to annotators change trust, corrective effort, and consistency in human-in-the-loop engagement annotation.

desk verdict The S3 framing manipulation is a useful idea, but the paper never measures annotation accuracy—it compares predictions to adjustments and calls the difference accuracy. read the letter →

arxiv 2502.07404 v2 pith:FSK2NIWK submitted 2025-02-11 cs.HC cs.AIcs.CV

Sahana Yadnakudige Subramanya , Ko Watanabe , Andreas Dengel , Shoya Ishimaru This is my paper · ORCID
classification cs.HCcs.AIcs.CV
keywords human-in-the-loopengagementestimationannotationaccuracymodelreliabilitycognitiveframingannotatortrustemotionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether, and how, the behavior of human annotators changes when an image-based engagement model is more or less reliable, and when its reliability is described negatively. It reports three scenarios with 29 participants: unaltered predictions (S1), fabricated errors (S2), and identical predictions framed as coming from an unreliable model (S3). The central result is that both actual reliability and mere framing move annotator behavior: reliable predictions produce high trust and minimal correction, unreliable predictions provoke larger and more variable corrections but also frustration, and negative framing alone increases skepticism and relabeling. The authors take this as evidence that human-in-the-loop frameworks should manage not only model quality but also the psychological conditions under which annotators see predictions.

What carries the argument

The central machinery is a three-scenario experimental comparison built on a MobileNetV2 + LSTM video model that outputs an engagement percentage, surrounded by a real-time human-in-the-loop relabeling interface. Participants saw each model prediction and could agree, disagree, or enter an adjusted engagement percentage. The argument is carried by comparing predicted and adjusted engagement distributions across scenarios using scatter plots, density plots, agreement bar charts, adjustment box plots, and the statistical battery (ANOVA, Kruskal-Wallis, chi-square, Cramér's V, Kolmogorov-Smirnov, Cohen's d). The S2 manipulation $|100-x|$ and the S3 framing instructions are the two levers that isolate actual reliability from perceived reliability.

What would settle it

Run the same three scenarios and compare every participant-adjusted engagement percentage with the DAiSEE expert gold-standard engagement label. If the mean absolute error of adjusted scores relative to gold standard is not lower than the raw model predictions' error, or if S2 and S3 adjustments are not closer to gold standard than S1 adjustments, then the claim that reliability and framing affect annotation accuracy would be contradicted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that two distinct factors change how people annotate engagement in a human-in-the-loop system. Actual prediction quality matters: when the model's engagement scores were correct, annotators agreed and adjusted little; when scores were deliberately distorted with the transformation $|100-x|$, annotators corrected more, disagreed more, and showed greater adjustment variability, with a Cohen's d of $-0.729$ between S1 and S2. Perceived quality matters independently: in S3 the same correct predictions were described as unreliable, and annotators showed moderate deviation, more relabeling, and a balanced mix of agreement and disagreement. Statistical tests (ANOVA, Kruskal-Wallis, chi-square with Cramér's V of 0.51, and KS tests) all indicate scenario differences. The paper concludes that model reliability and psychological framing significantly impact annotators' trust, engagement, and consistency, offering a lever for optimizing human-in-the-loop frameworks.

Load-bearing premise

The load-bearing premise is that the adjusted engagement percentages reflect annotation accuracy; the study measures how far participants moved from model predictions, but it never validates those adjustments against true engagement labels, so if corrections do not move toward ground truth the accuracy claims do not follow.

Editorial extensions

If this is right

  • Designers of human-in-the-loop annotation tools can change annotator vigilance through reliability messaging alone, without retraining the underlying model.
  • Reliable predictions can be used to keep annotator workload low, while deliberately flagging uncertainty could be reserved for cases where critical evaluation is actually needed.
  • Fabricated error rates that are too high risk frustration and variable annotations, so system designers need to calibrate how often a model is allowed to be wrong.
  • Negative framing produces more relabeling, so transparent communication about model quality is a design decision with measurable behavioral consequences.
  • The scenario differences give an empirical base for adaptive workflows that route predictions to humans based on both model confidence and user state.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims: a direct test of annotation accuracy would compare adjusted values to the DAiSEE gold-standard labels; the paper does not report this, so whether corrections improved accuracy is still open.
  • The S3 result implies that trust can be manipulated by narrative alone; if human-in-the-loop systems are deployed in high-stakes settings, calibration of reliability messages may be as consequential as model calibration.
  • One testable extension is to vary the framing message continuously (for instance, stated accuracy from 95% down to 50%) to map a dose-response curve of suspicion and correction behavior.
  • The finding that younger and frequent-AI participants relabel more suggests personalization of human-in-the-loop interfaces could be explored, though the small sample makes that association preliminary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a human-in-the-loop (HITL) annotation study in which participants adjusted engagement predictions produced by a MobileNetV2+LSTM model trained on the DAiSEE dataset. Three scenarios are compared: baseline reliability with unaltered predictions (S1), intentionally distorted predictions using |100−x| (S2), and identical predictions to S1 but with negative framing about model reliability (S3). The authors analyze predicted vs. adjusted engagement percentages through scatter/density/box plots, agreement bar charts, statistical tests (ANOVA, Kruskal-Wallis, chi-square, KS, Cohen's d), and qualitative feedback. They conclude that model reliability and psychological framing significantly impact annotators' trust, engagement, and consistency, and the title and abstract frame these findings as affecting 'annotation accuracy.'

Significance. The behavioral comparisons, especially S1 vs. S3, isolate the effect of perceived reliability while keeping predictions identical; if the effects replicate, they provide useful design guidance for HITL systems. The paper also contributes a working engagement-estimation model (F1 86% on engagement) and a real-time HITL annotation interface. However, the central accuracy claim is not supported by the current analyses, which never compare participant adjustments against the DAiSEE gold-standard labels. The strengths of the study are the within-study comparison across framing conditions and the transparent description of the S2 manipulation, but the lack of external validation limits the title's promise.

major comments (4)
  1. [§4.1–§4.2, Abstract, Title] The 'annotation accuracy' claim is unverified. All outcome analyses in Sections 4.1 and 4.2 compare the model's predicted engagement percentage with the participant's adjusted percentage; no analysis compares either quantity with the DAiSEE gold-standard labels, even though Section 3.1 states such labels exist. Consequently, larger adjustments in S2 and S3 are interpreted as 'corrections' or 'skepticism,' but the data only show deviation from the prediction, not movement toward the true engagement value. If participants adjusted away from the gold standard, the study would demonstrate that framing changes behavior while decreasing accuracy—an important but opposite result. The title and abstract should be revised, or the analysis should be supplemented with a gold-standard comparison, before the accuracy claim can be accepted.
  2. [§3.3, §5.3] The S2 manipulation |100 − x| is an inversion of the predicted scale rather than a realistic error distribution. This makes disagreement with the model's prediction a by-construction property of the manipulation: every prediction is exactly the complement of the original score, so participants are likely correcting an obvious mathematical transformation. The paper's own limitation in Section 5.3 acknowledges that 'S2's intentional errors could be extended to study how annotators respond to genuine error distributions,' which concedes the point. To support the claim that the study assesses responses to unreliable models, the error injection should mimic the actual error distribution of the model or at least be validated as plausible model output.
  3. [§3.3, §4.2, Table 2] The participant counts are inconsistent and the statistical treatment of repeated measures is unclear. The abstract states the study involved 29 participants, while Section 3.3 reports 33 participants in S1 and 29 in S2/S3, and the limitations section repeats both numbers. It is also unclear whether the same participants completed all three scenarios; no counterbalancing or order randomization is reported. The ANOVA, Kruskal-Wallis, and chi-square tests in Table 2 treat the scenario comparisons as independent, which would be inappropriate if the same participants contributed data to multiple scenarios. The authors should clarify the design (between- vs. within-subjects) and report the appropriate repeated-measures tests or mixed models.
  4. [§4.2, Table 2] The statistical reporting is incomplete. The Kolmogorov-Smirnov statistics are reported without p-values, and no multiple-comparison correction is described for the three pairwise comparisons (S1-S2, S1-S3, S2-S3) or for the family of tests. The ANOVA p-value of 0.0187, for example, may not survive a Bonferroni or Holm correction when considered alongside the other tests. Please report exact p-values, confidence intervals, and the correction method used.
minor comments (4)
  1. [Abstract and §3.3] The participant count is inconsistent: the abstract says '29 participants' but Section 3.3 reports '33 participants were recruited for S1' and 29 for S2/S3; please align these numbers.
  2. [Figure 2] The caption 'Sample Imaginative Question for S1' is unclear; please describe what makes the question imaginative and how it differs from the 'Puzzle Questions' in S2 and S3.
  3. [§4.1] The scatter, density, and box-plot interpretations (e.g., 'points are mostly clustered near the diagonal') are qualitative; consider adding quantitative summaries such as mean absolute adjustment per scenario to support these claims.
  4. [§5.2] The post-survey section reports age and gender trends (e.g., 'males as most labelers') without statistical tests; these claims should be either supported with appropriate analyses or explicitly framed as informal observations.

Circularity Check

1 steps flagged · score 3.0 of 10

Partially built-in S2 disagreement, but the study is an empirical manipulation and S3 provides independent non-circular evidence.

  1. self definitional [Section 3.3 (Data Collection), S2 definition; Section 4.1 (Scenario Based Analysis), S2 scatter/density interpretation; Table 2 statistics]
    "S2 Model predictions were intentionally distorted using the formula |100 − x|, where x represents the original engagement score. ... In S2 (Figure 3b), we see a significant shift away from the diagonal, as participants actively corrected the flawed predictions, highlighting their engagement and willingness to intervene."

    The S2 independent variable is defined as a deterministic inversion of the model's own score (p = 100 − x). Any participant who judges engagement near the true value x will therefore produce an adjusted value that differs from p by construction for nearly all x ≠ 50. The reported S2 outcomes — large deviations from the diagonal, dominant disagreement, and significant ANOVA/Kruskal-Wallis/Cohen's d effects — are consequently built into the input transformation rather than being an independent discovery about model reliability. The only genuinely empirical part is that participants did choose to correct rather than accept the inverted predictions; but the direction and size of the S1-vs-S2 difference in disagreement is predetermined by the formula.

full rationale

This paper is an experimental HCI study rather than a mathematical derivation, so most default circularity patterns do not apply. The model contribution (C1, MobileNetV2+LSTM, F1=86%) is evaluated on the DAiSEE test split with standard metrics, and no load-bearing argument depends on a self-citation: the authors' prior work appears only in background references. The S3 condition is the strongest independent evidence: predictions identical to S1 are paired with negative reliability framing, and participants adjust more and relabel more, so framing affects behavior even when model outputs are unchanged. The one partially built-in result is S2: the manipulation p=|100−x| guarantees that predictions are wrong in a maximally visible way, so the observed large disagreement and correction effects in Sections 4.1–4.2 are largely an artifact of the input transformation rather than a general discovery about 'model reliability'. In addition, the title's 'annotation accuracy' is never validated against DAiSEE gold-standard labels; all analyses compare predicted and adjusted percentages, so the accuracy wording is a construct-validity concern, not a circular derivation. On balance, the circular component is limited to the S2 operationalization, giving a score of 3.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All free parameters are model training hyperparameters or the S2 manipulation formula; none are fitted to the behavioral outcomes. The reported effects rest on four domain assumptions, chiefly that adjustments measure accuracy and that scenario order is not confounded.

free parameters (4)
  • learning_rate = 1e-4
    Chosen based on validation performance during model training (Section 3.2); not fitted to the behavioral data but a hand-chosen hyperparameter for the engagement model.
  • batch_size = 16
    Chosen for memory/stability tradeoff (Section 3.2).
  • max_epochs = 10
    Maximum epochs with early stopping (Section 3.2).
  • S2 error injection formula = |100 - x|
    Ad hoc transformation of original engagement score to simulate unreliable predictions (Section 3.3); this defines the S2 manipulation rather than being fit.
assumptions (4)
  • domain assumption DAiSEE ground-truth labels (crowdsourced, Dawid-Skene consolidated, expert-validated subset) are correct
    The model is trained and evaluated against these labels (Section 3.1); if the labels are wrong, the reported F1 and the reliability framing collapse.
  • domain assumption Participant-adjusted engagement percentages reflect annotation quality
    The study never compares adjustments to gold-standard labels; the outcome is treated as meaningful annotation behavior (Sections 4.1 and 4.2).
  • domain assumption Scenario order and stimulus identity do not confound cross-scenario differences
    No counterbalancing or order randomization is reported in Section 3.3; effects could be due to learning or fatigue.
  • domain assumption Statistical tests are applied to comparable repeated-measures or independent data as assumed
    The paper does not state whether ANOVA and Kruskal-Wallis treat scenarios as repeated measures or independent groups; the assumption is unverified (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy." pith.science (2026). https://pith.science/paper/FSK2NIWK

@misc{pith2026250207404,
  author       = {Pith},
  title        = {Pith review of: Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSK2NIWK}},
  note         = {Machine review of arXiv:2502.07404}
}
read the original abstract

Human-in-the-loop (HITL) frameworks are increasingly recognized for their potential to improve annotation accuracy in emotion estimation systems by combining machine predictions with human expertise. This study focuses on integrating a high-performing image-based emotion model into a HITL annotation framework to evaluate the collaborative potential of human-machine interaction and identify the psychological and practical factors critical to successful collaboration. Specifically, we investigate how varying model reliability and cognitive framing influence human trust, cognitive load, and annotation behavior in HITL systems. We demonstrate that model reliability and psychological framing significantly impact annotators' trust, engagement, and consistency, offering insights into optimizing HITL frameworks. Through three experimental scenarios with 29 participants--baseline model reliability (S1), fabricated errors (S2), and cognitive bias introduced by negative framing (S3)--we analyzed behavioral and qualitative data. Reliable predictions in S1 yielded high trust and annotation consistency, while unreliable outputs in S2 led to increased critical evaluations but also heightened frustration and response variability. Negative framing in S3 revealed how cognitive bias influenced participants to perceive the model as more relatable and accurate, despite misinformation regarding its reliability. These findings highlight the importance of both reliable machine outputs and psychological factors in shaping effective human-machine collaboration. By leveraging the strengths of both human oversight and automated systems, this study establishes a scalable HITL framework for emotion annotation and lays the foundation for broader applications in adaptive learning and human-computer interaction.

Figures

Figures reproduced from arXiv: 2502.07404 by the authors.

Figure 1
Figure 1. MobileNetV2 + LSTM Architecture was implemented to prevent overfitting. Class imbalance was addressed by cal￾culating class weights using the inverse frequency of each emotion, ensuring that less frequent emotional states contributed proportionally during training. Key hyperparameters included: – Batch Size: Set to 16 to balance memory efficiency with stable weight updates. – Epochs: A maximum of ten epochs with ear… view at source ↗
Figure 2
Figure 2. Comparison of Questions Across Scenarios [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Scatter plots comparing predicted and adjusted engagement percentages [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Density plots comparing the distributions of predicted and adjusted en [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Bar charts illustrating feedback agreement proportions across three sce [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Box plots depicting the variability in engagement adjustments across three [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages

  1. [1]

    ICIDSSD (2020)

    Ali,S.,Tanweer,S.,Khalid,S.S., Rao,N.:Melfrequencycepstralcoefficient: a review. ICIDSSD (2020)

    work page 2020

  2. [2]

    AI magazine35(4), 105–120 (2014)

    Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: The role of humans in interactive machine learning. AI magazine35(4), 105–120 (2014)

    work page 2014

  3. [3]

    Proceedings of the Romanian Academy11(3), 277–283 (2010)

    Barbu, T.: Gabor filter-based face recognition technique. Proceedings of the Romanian Academy11(3), 277–283 (2010)

    work page 2010

  4. [4]

    Wiley stat- sref: Statistics reference online (2014)

    Berger, V.W., Zhou, Y.: Kolmogorov–smirnov test: Overview. Wiley stat- sref: Statistics reference online (2014)

    work page 2014

  5. [5]

    Journal of the Korean Statistical Society42(3), 323–328 (2013)

    Bergsma, W.: A bias-correction for cramér’s v and tschuprow’s t. Journal of the Korean Statistical Society42(3), 323–328 (2013)

    work page 2013

  6. [6]

    IEEE Access 12, 192219–192229 (2024)

    Bhatt, A., Watanabe, K., Santhosh, J., Dengel, A., Ishimaru, S.: Estimating self-confidence in video-based learning using eye-tracking and deep neural networks. IEEE Access 12, 192219–192229 (2024). https://doi.org/10.1109/ACCESS.2024.3515838

    work page arXiv 2024

  7. [7]

    In: Proceedings of the 20th international conference on intelligent user interfaces

    Bosch, N., D’Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., Zhao, W.: Automatic detection of learning-centered affec- tive states in the wild. In: Proceedings of the 20th international conference on intelligent user interfaces. pp. 379–388 (2015)

    work page 2015

  8. [8]

    Sensors & Materials34 (2022)

    Chen, C., Arakawa, Y., Watanabe, K., Ishimaru, S.: Quantitative evaluation system for online meetings based on multimodal microbehavior analysis. Sensors & Materials34 (2022)

    work page 2022

Show all 47 references
  1. [9]

    Scientific Reports12(1), 6991 (2022)

    Debnath, T., Reza, M.M., Rahman, A., Beheshti, A., Band, S.S., Alinejad- Rokny, H.: Four-layer convnet to facial emotion recognition with minimal epochs and the significance of data diversity. Scientific Reports12(1), 6991 (2022)

    2022

  2. [10]

    ACM Transactions on Interactive Intelligent Systems (TiiS)2(4), 1– 39 (2013)

    D’mello, S., Graesser, A.: Autotutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS)2(4), 1– 39 (2013)

    2013

  3. [11]

    Pattern recognition 44(3), 572–587 (2011)

    El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recog- nition: Features, classification schemes, and databases. Pattern recognition 44(3), 572–587 (2011)

    2011

  4. [12]

    Psychological bulletin128(2), 203 (2002)

    Elfenbein, H.A., Ambady, N.: On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological bulletin128(2), 203 (2002)

    2002

  5. [13]

    Multimedia Tools and Applications pp

    Erat, K., Şahin, E.B., Doğan, F., Merdanoğlu, N., Akcakaya, A., Durdu, P.O.: Emotion recognition with eeg-based brain-computer interfaces: a sys- tematic literature review. Multimedia Tools and Applications pp. 1–48 (2024) 18 Subramanya and Watanabe et al

    2024

  6. [14]

    IEEE Transactions on Computational Social Systems (2023)

    Gong, X., Chen, C.P., Zhang, T.: Cross-cultural emotion recognition with eeg and eye movement signals based on multiple stacked broad learning system. IEEE Transactions on Computational Social Systems (2023)

    2023

  7. [15]

    arXiv preprint arXiv:1609.01885 (2016)

    Gupta, A., D’Cunha, A., Awasthi, K., Balasubramanian, V.: Daisee: Towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885 (2016)

    2016 arXiv

  8. [16]

    Computers & Education 90, 36–53 (2015)

    Henrie, C.R., Halverson, L.R., Graham, C.R.: Measuring student engage- ment in technology-mediated learning: A review. Computers & Education 90, 36–53 (2015)

    2015

  9. [17]

    Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., Wallach, H.: Improving fairness in machine learning systems: What do industry practi- tioners need? In: Proceedings of the 2019 CHI conference on human factors in computing systems. pp. 1–16 (2019)

    2019

  10. [18]

    In: Proceedings of the 11th ACM Symposium on Eye Tracking Re- search & Applications

    John, B.: Pupil diameter as a measure of emotion and sickness in vr. In: Proceedings of the 11th ACM Symposium on Eye Tracking Re- search & Applications. ETRA ’19, Association for Computing Machin- ery, New York, NY, USA (2019). https://doi.org/10.1145/3314111.3322868, https:/...

    2019

  11. [19]

    Technology, Knowledge and Learning 27(2), 449–460 (2022)

    Karaoglan Yilmaz, F.G., Yilmaz, R.: Learning analytics intervention im- proves students’ engagement in online learning. Technology, Knowledge and Learning 27(2), 449–460 (2022)

    2022

  12. [20]

    In: 2016 Asia-Pacific signal and information processing association annual summit and conference (APSIPA)

    Lim, W., Jang, D., Lee, T.: Speech emotion recognition using convolutional and recurrent neural networks. In: 2016 Asia-Pacific signal and information processing association annual summit and conference (APSIPA). pp. 1–4. IEEE (2016)

    2016

  13. [21]

    Scientific Reports13(1), 8414 (2023)

    Lukac, M., Zhambulova, G., Abdiyeva, K., Lewis, M.: Study on emotion recognition bias in different regional groups. Scientific Reports13(1), 8414 (2023)

    2023

  14. [22]

    In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    Luo, Y., Yu, J., Liang, M., Wan, Y., Zhu, K., Santosa, S.S.: Emo- tion embodied: Unveiling the expressive potential of single-hand ges- tures. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. CHI ’24, Association for Computing Machinery, New Yo...

    2024

  15. [23]

    Markus,H.R.,Kitayama,S.:Cultureandtheself:Implicationsforcognition, emotion,andmotivation.In:Collegestudentdevelopmentandacademiclife, pp. 264–293. Routledge (2014)

    2014

  16. [24]

    Sensors18(11), 3978 (2018)

    Matsuda, Y., Fedotov, D., Takahashi, Y., Arakawa, Y., Yasumoto, K., Minker, W.: Emotour: Estimating emotion and satisfaction of users based on behavioral cues and audiovisual data. Sensors18(11), 3978 (2018)

    2018

  17. [25]

    Biochemia medica 23(2), 143–149 (2013)

    McHugh, M.L.: The chi-square test of independence. Biochemia medica 23(2), 143–149 (2013)

    2013

  18. [26]

    The corsini encyclopedia of psychology pp

    McKight, P.E., Najab, J.: Kruskal-wallis test. The corsini encyclopedia of psychology pp. 1–1 (2010)

    2010

  19. [27]

    Frontiers in Robotics and AI8, 555913 (2021) Human-in-the-Loop Annotation for Image-Based Engagement Estimation 19

    Oertel, C., Jonell, P., Kontogiorgos, D., Mora, K.F., Odobez, J.M., Gustafson, J.: Towards an engagement-aware attentive artificial listener for multi-party interactions. Frontiers in Robotics and AI8, 555913 (2021) Human-in-the-Loop Annotation for Image-Based Engagement Estimation 19

    2021

  20. [28]

    The Journal of Experimental Education82(1), 22–50 (2014)

    Peng, C.Y.J., Chen, L.T.: Beyond cohen’s d: Alternative effect size measures for between-subject designs. The Journal of Experimental Education82(1), 22–50 (2014)

    2014

  21. [29]

    arXiv preprint arXiv:2209.15370 (2022)

    Salam, H., Celiktutan, O., Gunes, H., Chetouani, M.: Automatic context- driven inference of engagement in hmi: A survey. arXiv preprint arXiv:2209.15370 (2022)

    2022 arXiv

  22. [30]

    Sharma, K., Jermann, P., Dillenbourg, P.: Displaying teacher’s gaze in a mooc: Effects on students’ video navigation patterns. In: Design for Teach- ing and Learning in a Networked World: 10th European Conference on Technology Enhanced Learning, EC-TEL 2015, Toledo, Spain, Sep...

    2015

  23. [31]

    Sensors18(7), 2074 (2018)

    Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., Yang, X.: A review of emotion recognition using physiological signals. Sensors18(7), 2074 (2018)

    2018

  24. [32]

    Chemometrics and intelligent laboratory systems6(4), 259–272 (1989)

    St, L., Wold, S., et al.: Analysis of variance (anova). Chemometrics and intelligent laboratory systems6(4), 259–272 (1989)

    1989

  25. [33]

    IEEE Internet of Things Journal (2023)

    Taherisadr,M.,AlFaruque,M.A.,Elmalaki,S.:Erudite:Human-in-the-loop iot for an adaptive personalized learning system. IEEE Internet of Things Journal (2023)

    2023

  26. [34]

    In: Companion of the 2024 on ACM International Joint Conference on Per- vasive and Ubiquitous Computing

    Tanaka, N., Watanabe, K., Ishimaru, S., Dengel, A., Ata, S., Fujimoto, M.: Concentration estimation in online video lecture using multimodal sensors. In: Companion of the 2024 on ACM International Joint Conference on Per- vasive and Ubiquitous Computing. pp. 71–75 (2024)

    2024

  27. [35]

    In: Affective Computing for Social Good: Enhancing Well-being, Empathy, and Equity, pp

    Vairamani, A.D.: Advancements in multimodal emotion recognition: Inte- grating facial expressions and physiological signals. In: Affective Computing for Social Good: Enhancing Well-being, Empathy, and Equity, pp. 217–240. Springer (2024)

    2024

  28. [36]

    Neurocomputing312, 251–262 (2018)

    Wang, S.J., Li, B.J., Liu, Y.J., Yan, W.J., Ou, X., Huang, X., Xu, F., Fu, X.:Micro-expressionrecognitionwithsmallsamplesizebytransferringlong- term convolutional neural network. Neurocomputing312, 251–262 (2018)

    2018

  29. [37]

    Frontiers in psychology14, 1126994 (2023)

    Wang, X., Ren, Y., Luo, Z., He, W., Hong, J., Huang, Y.: Deep learning- based eeg emotion recognition: Current trends and future perspectives. Frontiers in psychology14, 1126994 (2023)

    2023

  30. [38]

    In: Proceedings of the Augmented Humans International Conference 2024

    Watanabe, K., Dengel, A., Ishimaru, S.: Metacognition-engauge: Real-time augmentation of self-and-group engagement levels understanding by gauge interface in online meetings. In: Proceedings of the Augmented Humans International Conference 2024. pp. 301–303 (2024)

    2024

  31. [39]

    IEEE Access11, 52886–52898 (2023)

    Watanabe, K., Sathyanarayana, T., Dengel, A., Ishimaru, S.: Engauge: En- gagement gauge of meeting participants estimated by facial expression and deep neural network. IEEE Access11, 52886–52898 (2023)

    2023

  32. [40]

    Sensors21(17), 5719 (2021)

    Watanabe, K., Soneda, Y., Matsuda, Y., Nakamura, Y., Arakawa, Y., Den- gel, A., Ishimaru, S.: Discaas: Micro behavior analysis on discussion by camera as a sensor. Sensors21(17), 5719 (2021)

    2021

  33. [41]

    British Journal of Educational Technology47(6), 1304–1323 (2016) 20 Subramanya and Watanabe et al

    Wu, C.H., Huang, Y.M., Hwang, J.P.: Review of affective computing in education/learning: Trends and challenges. British Journal of Educational Technology47(6), 1304–1323 (2016) 20 Subramanya and Watanabe et al

    2016

  34. [42]

    Future Generation Computer Systems135, 364–381 (2022)

    Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., He, L.: A survey of human-in- the-loop for machine learning. Future Generation Computer Systems135, 364–381 (2022)

    2022

  35. [43]

    IEEE Access7, 108906–108915 (2019)

    Yan, K., Zheng, W., Zhang, T., Zong, Y., Tang, C., Lu, C., Cui, Z.: Cross- domain facial expression recognition based on transductive deep transfer learning. IEEE Access7, 108906–108915 (2019)

    2019

  36. [44]

    In: 2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)

    Yigitbas, E., Karakaya, K., Jovanovikj, I., Engels, G.: Enhancing human- in-the-loop adaptive systems through digital twins and vr interfaces. In: 2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). pp. 30–40. IEEE (2021)

    2021

  37. [45]

    Neural Com- puting and Applications pp

    Younis, E.M., Mohsen, S., Houssein, E.H., Ibrahim, O.A.S.: Machine learn- ing for human emotion recognition: a comprehensive review. Neural Com- puting and Applications pp. 1–47 (2024)

    2024

  38. [46]

    In: Proceedings of the 9th international conference on Multimodal interfaces

    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recog- nition methods: audio, visual and spontaneous expressions. In: Proceedings of the 9th international conference on Multimodal interfaces. pp. 126–133 (2007)

    2007

  39. [47]

    In: 2023 IEEE 16th Pacific Visualization Symposium (PacificVis)

    Zhang, X., Xuan, X., Dima, A., Sexton, T., Ma, K.L.: Labelvizier: Interac- tive validation and relabeling for technical text annotations. In: 2023 IEEE 16th Pacific Visualization Symposium (PacificVis). pp. 167–176. IEEE (2023)

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.