REVIEW 3 major objections 6 minor 23 references
On ambivalence/hesitancy video recognition, honest calibration beats fancy fusion: AP-weighted averaging at a fixed 0.5 threshold lifts public-test macro-F1 from 0.690 to 0.731.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
On BAH, language and ASR-erased timing features dominate A/H detection; AP-weighted ensembles at a fixed 0.5 threshold reach 0.731 macro-F1 while validation-tuned calibration overfits.
T0 review reviewed 2026-07-14 challenge →
load-bearing objection Solid challenge paper: ASR-erased time is a real, under-used signal and the calibration ablation is the part worth keeping; scope is one small BAH dataset. the 3 major comments →
Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the BAH ambivalence/hesitancy task, calibration is the dominant lever. Fitting ensemble weights and a decision threshold on the 124-video validation split scores 0.741 macro-F1 on validation but only 0.690 on the untouched public test; AP-weighted probability averaging at a fixed threshold of 0.5 instead reaches 0.731, above the previous edition’s winner. Language is by far the strongest channel; conflict design does not reliably help; and ASR-erased time is the strongest independent non-verbal signal found.
What carries the argument
Affective Marker Fusion (AMF) plus AP-weighted ensemble at fixed threshold: each stream is projected, then multiplied by a learned reliability gate so weak channels are down-weighted; six members are finally combined by average-precision weights with the decision threshold locked at 0.5. ASR-erased time supplies an independent 16-dimensional non-verbal channel recovered from timestamp gaps left when ASR deletes fillers and pauses.
Load-bearing premise
That average precision measured on a small held-out set (124 or 113 videos) is stable enough to set ensemble weights that transfer to the public test, even when that same holdout is also used for early stopping.
What would settle it
Retrain the identical six members and recompute AP weights on a larger, never-before-seen participant-wise holdout; if the public-test macro-F1 then falls back near 0.690 or the AP weights become unstable, the calibration claim fails.
If this is right
- Future BAH systems should freeze the decision threshold and weight members by a threshold-free ranking metric rather than searching both on the validation split.
- Language and a few readable hesitation markers can nearly match a full multimodal pipeline, so text-first baselines become the natural starting point.
- ASR-erased time is available for free from any timestamped transcript and is nearly uncorrelated with existing members, so it is a ready orthogonal feature for larger data regimes.
- Hand-designed cross-modal conflict operators can be dropped without loss once affect-specialised encoders are in place.
Where Pith is reading between the lines
- The same calibration trap—searching weights and thresholds on a tiny validation split—is likely costing points in other small-data multimodal affect challenges that report only validation numbers.
- Because the strongest non-verbal signal is the time ASR erases rather than what the face or voice models extract, future work may profit more from better timestamp recovery than from larger visual backbones.
- If more labelled data become available, ASR-erased time’s independence should convert into accuracy gains that the current 778-video regime cannot yet realise.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper tackles binary ambivalence/hesitancy (A/H) recognition on the ABAW 2026 BAH dataset. It combines frozen affect-specialised encoders (RoBERTa-GoEmotions text, emotion wav2vec2 audio, FER-ViT video) with eleven hand-crafted linguistic hesitation markers, fused via a reliability-gated module (Affective Marker Fusion, AMF), and finished with an AP-weighted probability ensemble at a fixed threshold τ=0.5. A second contribution is ASR-erased time: sixteen features recovered from Whisper chunk-timestamp gaps left by deleted fillers and pauses. Controlled ablations support three claims: (i) cross-modal conflict operators (absolute difference, orthogonal split, or none) do not reliably improve public-test performance once encoders are fixed (Table 5); (ii) language dominates (text AP 0.811; markers AP 0.800), with emotion audio a useful second and all visual streams weak (Table 4); (iii) calibration dominates architecture—searching ensemble weights and τ on the 124-video validation split yields 0.741 val / 0.690 test macro-F1, while AP weights at fixed τ=0.5 reach 0.731 on the same public test (Table 6), above the prior winner (0.694).
Significance. If the calibration result holds under broader scrutiny, the paper supplies a clear, transferable lesson for small-data multimodal affect: threshold-free AP weighting at a fixed decision boundary can outperform validation-tuned weights and thresholds by several macro-F1 points, and this gain can exceed architectural choices such as conflict operators. The controlled encoder-fixed ablations (Tables 4–5) and the explicit overfit signature (0.741→0.690) are stronger evidence than is typical in challenge reports. ASR-erased time is a genuinely clever, nearly uncorrelated non-verbal channel (AP 0.718; correlations 0.11–0.36). The pipeline is deterministic, one-command, and code-linked, which is a concrete reproducibility strength. The absolute public-test margin over 0.694 is modest and the bootstrap interval [0.693, 0.770] has a lower edge that touches the prior winner, so the ranking claim should be read as probable rather than definitive; the methodological message about honest calibration remains the more durable contribution.
major comments (3)
- Table 6 confounds the calibration recipe with ensemble size: the searched-weights+τ row uses 4 members (test F1 0.690) while the AP-weight rows use 5 and 6 members (0.727 / 0.731). The 4-point gap attributed to calibration therefore partly reflects membership. A matched-size ablation (same 4 or same 6 members under both recipes) is needed to isolate the claimed effect; without it the central “calibration > architecture” claim is only partially controlled.
- Sec. 3.4 introduces ASR-erased time as a core contribution (16 features; strongest non-verbal channel) but only lists categories (gap counts/sizes, speaking-rate variability, pre-contrast gaps, restarts). Exact definitions, normalisation, and the monotone-timeline rule are not given with sufficient precision for independent reimplementation. Because the feature set is claimed to be novel and nearly uncorrelated with all model members, a complete, deterministic specification (or a pointer to the released code that enumerates them) is load-bearing for the contribution.
- Secs. 3.6 and 4.7 dual-use the same 113-video stratified holdout for early stopping and for AP weight estimation when training on all labelled data. The authors correctly flag that holdout member scores are mildly optimistic, yet the final ensemble weights that produce the locked private-test predictions still depend on that dual use. A nested or repeated holdout (or leave-one-participant-out AP) would quantify how much the reported 0.731-class public-test figure and the private submission depend on this coupling; the current design leaves residual contamination risk on the all-data track.
minor comments (6)
- Fig. 1 caption states that gaze/brow, prosody, and cue supervision are excluded because they hurt the model; Sec. 4.8 quantifies this only for gaze/brow (AP 0.860→0.841). A one-line table or parenthetical for prosody and cue-supervision drops would make the exclusion claim fully checkable.
- Eq. (2) writes BCEw(p,y)=-,w,ylogp-(1-y)log(1-p); the leading comma after the minus sign is a typesetting error and the positive-class weight w is applied only to the positive term, which should be stated explicitly.
- Table 7 reports AMF holdout F1 0.751 below several encoder members, yet the public-test single-member AMF is 0.725 (Table 3) and is kept because “the leak-free public test favours it.” A short note reconciling holdout vs public-test ranking of AMF would reduce reader confusion.
- The abstract and Sec. 1 state “sixteen features” for ASR-erased time and “eleven” hesitation markers; Sec. 3.3 later refers to a “25-dim marker pathway.” Clarify whether 25 = 11 + 14 (FER stats) or another concatenation so dimensions stay consistent.
- Related Work cites the previous winner as 0.694 public / 0.715 private [2]; the private number is not used elsewhere. Either drop it or note that private scores are unavailable for the present submission so the comparison is public-only.
- Minor prose: “ABA W” with a space appears once in Sec. 2; “co-attention baseline 0.59” should be 0.590 for consistency with Table 3.
Circularity Check
No circularity: empirical ablations and held-out test comparisons; AP weights and fixed threshold are not forced by construction to the reported public-test gains.
full rationale
This is a controlled empirical systems paper on BAH video classification. Its three organising claims (conflict operators do not reliably help; language dominates; AP-weighted averaging at fixed τ=0.5 outperforms validation-tuned weights+threshold) are established by direct measurement on the untouched public test (525 videos) or by ablations that hold encoders/training fixed and vary only one factor (Tables 4–6). Ensemble weights are set by holdout AP (Eq. 3) and the threshold is deliberately fixed at 0.5 precisely to avoid the overfitting the authors themselves document (0.741 val → 0.690 test). The load-bearing 0.731 figure is therefore not a fitted quantity renamed as a prediction; it is an out-of-sample score. Dual use of the 113-video holdout for early stopping and weighting is flagged by the authors as producing mildly optimistic member numbers, but that does not make the public-test comparison circular. No self-definitional equations, no uniqueness theorems imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result appear. Citations are to external encoders, datasets and prior ABAW baselines. The derivation chain is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (7)
- marker-head auxiliary loss weight =
0.3
- label smoothing epsilon =
0.1
- AMF shared projection width =
256
- decision threshold tau =
0.5
- ensemble member AP weights =
normalized holdout APs
- class-balance positive weight w =
inverse positive frequency
- learning rates (text vs other heads) =
2e-5 / 1e-4
axioms (5)
- domain assumption Frozen affect-specialised encoders (RoBERTa-GoEmotions, FER-ViT, emotion wav2vec2, plus VideoMAE/HuBERT members) plus lightweight trained heads are sufficient to expose the BAH signal.
- domain assumption BAH participant-wise splits yield independent train/val/public-test evaluations with reliable binary A/H labels.
- ad hoc to paper Average precision on a small holdout is a stable, threshold-free ranking signal for ensemble weights that transfer better than F1-tuned weights and thresholds.
- domain assumption Whisper (or equivalent) chunk timestamps remain monotone within 30 s segments so gap features are well-defined after filler deletion.
- standard math Standard binary cross-entropy with class reweighting and label smoothing is an adequate training objective for noisy video-level A/H labels.
invented entities (2)
-
Affective Marker Fusion (AMF)
no independent evidence
-
ASR-erased time (16-d feature set)
no independent evidence
Cite this review
Pith. "Pith review of Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video." pith.science (2026). https://pith.science/paper/IC6SL67D
@misc{pith2026260711120,
author = {Pith},
title = {Pith review of: Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video},
year = {2026},
howpublished = {\url{https://pith.science/paper/IC6SL67D}},
note = {Machine review of arXiv:2607.11120}
}
abstract
We address ambivalence and hesitancy (A/H) recognition in the ABAW 2026 BAH Challenge: given a short interview video, predict whether the person shows signs of A/H. Our system combines affect-specialised text, audio, and visual representations with a small set of readable linguistic hesitation cues, fused by a reliability gate we call Affective Marker Fusion (AMF), and finished with a simple AP-weighted ensemble at a fixed decision threshold. We also introduce \emph{ASR-erased time}: speech recognisers delete fillers and hesitation pauses from the transcript, but the chunk timestamps keep the time those events took, and sixteen features built from these gaps form the strongest and most independent non-verbal channel we measured (AP $0.718$, correlation $0.11$--$0.36$ with all other members). Across controlled experiments we find three things: cross-modal conflict design does not reliably help on BAH; language is by far the strongest channel while affect-specialised audio is a useful second; and calibration matters more than architecture. Fitting ensemble weights and a threshold on the small validation split overfits: it scores $0.741$ macro-F1 on validation but only $0.690$ on the untouched test set. AP-weighting at a fixed threshold instead reaches $\mathbf{0.731}$ on test.
Figures
Reference graph
Works this paper leans on
-
[1]
Bah dataset for am- bivalence/hesitancy recognition in videos for behavioural change,
M. Gonz ´alez-Gonz´alez, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger, “Bah dataset for am- bivalence/hesitancy recognition in videos for behavioural change,”arXiv preprint arXiv:2505.19328, vol. 3, no. 9, 2025
Pith/arXiv arXiv 2025
-
[2]
Conflict-aware multi- modal fusion for ambivalence and hesitancy recognition,
S. E. Bekhouche, H. Telli, A. Benlamoudi, S. E. Herrouz, A. Taleb-Ahmed, and A. Hadid, “Conflict-aware multi- modal fusion for ambivalence and hesitancy recognition,” arXiv preprint arXiv:2603.15818, 2026
arXiv 2026
-
[3]
A. V . Savchenko, “Hsemotion team at abaw-8 com- petition: Audiovisual ambivalence/hesitancy, emotional mimicry intensity and facial expression recognition,” arXiv preprint arXiv:2503.10399, 2025
Pith/arXiv arXiv 2025
-
[4]
The 6th affec- tive behavior analysis in-the-wild (abaw) competition,
D. Kollias, P. Tzirakis, A. Cowen, S. Zafeiriou, I. Kotsia, A. Baird, C. Gagne, C. Shao, and G. Hu, “The 6th affec- tive behavior analysis in-the-wild (abaw) competition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4587–4598, 2024
2024
-
[5]
Context-dependent sen- timent analysis in user-generated videos,
S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P. Morency, “Context-dependent sen- timent analysis in user-generated videos,” inProceedings of the 55th annual meeting of the association for com- putational linguistics (volume 1: Long papers), pp. 873– 883, 2017
2017
-
[6]
Verbal and nonverbal clues for real-life deception detection,
V . P´erez-Rosas, M. Abouelenien, R. Mihalcea, Y . Xiao, C. Linton, and M. Burzo, “Verbal and nonverbal clues for real-life deception detection,” inProceedings of the 2015 conference on empirical methods in natural lan- guage processing, pp. 2336–2346, 2015
2015
-
[7]
Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training,
Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training,”Advances in neural infor- mation processing systems, vol. 35, pp. 10078–10093, 2022
2022
-
[8]
Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units,
W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units,”IEEE/ACM transactions on au- dio, speech, and language processing, vol. 29, pp. 3451– 3460, 2021
2021
-
[9]
Dawn of the transformer era in speech emotion recog- nition: closing the valence gap,
J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recog- nition: closing the valence gap,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 10745–10759, 2023
2023
-
[10]
Roberta: A robustly optimized bert pretraining ap- proach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining ap- proach,”arXiv preprint arXiv:1907.11692, 2019
Pith/arXiv arXiv 1907
-
[11]
Goemotions: A dataset of fine-grained emotions,
D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “Goemotions: A dataset of fine-grained emotions,” inProceedings of the 58th annual meeting of the association for computational linguistics, pp. 4040–4054, 2020
2020
-
[12]
Deberta: Decoding-enhanced bert with disentangled attention,
P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv preprint arXiv:2006.03654, 2020
Pith/arXiv arXiv 2006
-
[13]
On calibration of modern neural networks,
C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInterna- tional Conference on Machine Learning, pp. 1321–1330, PMLR, 2017
2017
-
[14]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly,et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020
Pith/arXiv arXiv 2010
-
[15]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,”Advances in neural informa- tion processing systems, vol. 33, pp. 12449–12460, 2020
2020
-
[16]
Rethinking the inception architecture for computer vision,
C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wo- jna, “Rethinking the inception architecture for computer vision,” inProceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2818–2826, 2016
2016
-
[17]
Robust speech recog- nition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recog- nition via large-scale weak supervision,” inInterna- tional conference on machine learning, pp. 28492–28518, PMLR, 2023
2023
-
[18]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,”arXiv preprint arXiv:1711.05101, 2017. 5
Pith/arXiv arXiv 2017
-
[19]
R-drop: Regularized dropout for neural networks,
L. Wu, J. Li, Y . Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, T.-Y . Liu,et al., “R-drop: Regularized dropout for neural networks,”Advances in neural information processing systems, vol. 34, pp. 10890–10905, 2021
2021
-
[20]
Pytorch: An imperative style, high-performance deep learning library,
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019
2019
-
[21]
Transformers: State-of-the-art natural language process- ing,
T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz,et al., “Transformers: State-of-the-art natural language process- ing,” inProceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, 2020
2020
-
[22]
Efron and R
B. Efron and R. J. Tibshirani,An introduction to the boot- strap. Chapman and Hall/CRC, 1994
1994
-
[23]
Mediapipe: A frame- work for building perception pipelines,
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee,et al., “Mediapipe: A frame- work for building perception pipelines,”arXiv preprint arXiv:1906.08172, 2019. 6
Pith/arXiv arXiv 1906
This paper was first reviewed by grok-4.5 on July 14, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.