Pith. sign in

REVIEW 3 major objections 6 minor 23 references

On ambivalence/hesitancy video recognition, honest calibration beats fancy fusion: AP-weighted averaging at a fixed 0.5 threshold lifts public-test macro-F1 from 0.690 to 0.731.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

On BAH, language and ASR-erased timing features dominate A/H detection; AP-weighted ensembles at a fixed 0.5 threshold reach 0.731 macro-F1 while validation-tuned calibration overfits.

T0 review reviewed 2026-07-14 challenge →

load-bearing objection Solid challenge paper: ASR-erased time is a real, under-used signal and the calibration ablation is the part worth keeping; scope is one small BAH dataset. the 3 major comments →

arxiv 2607.11120 v1 pith:IC6SL67D submitted 2026-07-13 cs.CV cs.CLeess.AS

Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video

classification cs.CV cs.CLeess.AS
keywords ambivalence/hesitancy recognitionmultimodal affectensemble calibrationASR-erased timeAffective Marker Fusionhesitation markersBAH datasetABAW
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a binary video task: given a short interview clip, decide whether the speaker shows ambivalence or hesitancy. The authors build a multimodal system around affect-specialised text, audio and face features plus a handful of readable linguistic hesitation cues, fused through a reliability gate they call Affective Marker Fusion, then combined by a simple ensemble. Their controlled experiments yield three clear findings. First, hand-designed cross-modal conflict operators do not reliably help once the encoders are fixed. Second, almost all of the useful signal lives in language (a fine-tuned emotion text model and eleven simple hesitation markers), with emotion-specialised audio a useful second channel and every visual representation remaining weak. Third, and largest, how you calibrate the ensemble matters more than architecture: searching weights and a decision threshold on the small validation split overfits badly, while weighting members by average precision and locking the threshold at 0.5 recovers a public-test macro-F1 of 0.731. They also introduce ASR-erased time—sixteen features recovered from the gaps that speech recognisers leave when they delete fillers and pauses—which is the strongest non-verbal channel they measured and nearly uncorrelated with every other member.

Core claim

On the BAH ambivalence/hesitancy task, calibration is the dominant lever. Fitting ensemble weights and a decision threshold on the 124-video validation split scores 0.741 macro-F1 on validation but only 0.690 on the untouched public test; AP-weighted probability averaging at a fixed threshold of 0.5 instead reaches 0.731, above the previous edition’s winner. Language is by far the strongest channel; conflict design does not reliably help; and ASR-erased time is the strongest independent non-verbal signal found.

What carries the argument

Affective Marker Fusion (AMF) plus AP-weighted ensemble at fixed threshold: each stream is projected, then multiplied by a learned reliability gate so weak channels are down-weighted; six members are finally combined by average-precision weights with the decision threshold locked at 0.5. ASR-erased time supplies an independent 16-dimensional non-verbal channel recovered from timestamp gaps left when ASR deletes fillers and pauses.

Load-bearing premise

That average precision measured on a small held-out set (124 or 113 videos) is stable enough to set ensemble weights that transfer to the public test, even when that same holdout is also used for early stopping.

What would settle it

Retrain the identical six members and recompute AP weights on a larger, never-before-seen participant-wise holdout; if the public-test macro-F1 then falls back near 0.690 or the AP weights become unstable, the calibration claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Future BAH systems should freeze the decision threshold and weight members by a threshold-free ranking metric rather than searching both on the validation split.
  • Language and a few readable hesitation markers can nearly match a full multimodal pipeline, so text-first baselines become the natural starting point.
  • ASR-erased time is available for free from any timestamped transcript and is nearly uncorrelated with existing members, so it is a ready orthogonal feature for larger data regimes.
  • Hand-designed cross-modal conflict operators can be dropped without loss once affect-specialised encoders are in place.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same calibration trap—searching weights and thresholds on a tiny validation split—is likely costing points in other small-data multimodal affect challenges that report only validation numbers.
  • Because the strongest non-verbal signal is the time ASR erases rather than what the face or voice models extract, future work may profit more from better timestamp recovery than from larger visual backbones.
  • If more labelled data become available, ASR-erased time’s independence should convert into accuracy gains that the current 778-video regime cannot yet realise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper tackles binary ambivalence/hesitancy (A/H) recognition on the ABAW 2026 BAH dataset. It combines frozen affect-specialised encoders (RoBERTa-GoEmotions text, emotion wav2vec2 audio, FER-ViT video) with eleven hand-crafted linguistic hesitation markers, fused via a reliability-gated module (Affective Marker Fusion, AMF), and finished with an AP-weighted probability ensemble at a fixed threshold τ=0.5. A second contribution is ASR-erased time: sixteen features recovered from Whisper chunk-timestamp gaps left by deleted fillers and pauses. Controlled ablations support three claims: (i) cross-modal conflict operators (absolute difference, orthogonal split, or none) do not reliably improve public-test performance once encoders are fixed (Table 5); (ii) language dominates (text AP 0.811; markers AP 0.800), with emotion audio a useful second and all visual streams weak (Table 4); (iii) calibration dominates architecture—searching ensemble weights and τ on the 124-video validation split yields 0.741 val / 0.690 test macro-F1, while AP weights at fixed τ=0.5 reach 0.731 on the same public test (Table 6), above the prior winner (0.694).

Significance. If the calibration result holds under broader scrutiny, the paper supplies a clear, transferable lesson for small-data multimodal affect: threshold-free AP weighting at a fixed decision boundary can outperform validation-tuned weights and thresholds by several macro-F1 points, and this gain can exceed architectural choices such as conflict operators. The controlled encoder-fixed ablations (Tables 4–5) and the explicit overfit signature (0.741→0.690) are stronger evidence than is typical in challenge reports. ASR-erased time is a genuinely clever, nearly uncorrelated non-verbal channel (AP 0.718; correlations 0.11–0.36). The pipeline is deterministic, one-command, and code-linked, which is a concrete reproducibility strength. The absolute public-test margin over 0.694 is modest and the bootstrap interval [0.693, 0.770] has a lower edge that touches the prior winner, so the ranking claim should be read as probable rather than definitive; the methodological message about honest calibration remains the more durable contribution.

major comments (3)
  1. Table 6 confounds the calibration recipe with ensemble size: the searched-weights+τ row uses 4 members (test F1 0.690) while the AP-weight rows use 5 and 6 members (0.727 / 0.731). The 4-point gap attributed to calibration therefore partly reflects membership. A matched-size ablation (same 4 or same 6 members under both recipes) is needed to isolate the claimed effect; without it the central “calibration > architecture” claim is only partially controlled.
  2. Sec. 3.4 introduces ASR-erased time as a core contribution (16 features; strongest non-verbal channel) but only lists categories (gap counts/sizes, speaking-rate variability, pre-contrast gaps, restarts). Exact definitions, normalisation, and the monotone-timeline rule are not given with sufficient precision for independent reimplementation. Because the feature set is claimed to be novel and nearly uncorrelated with all model members, a complete, deterministic specification (or a pointer to the released code that enumerates them) is load-bearing for the contribution.
  3. Secs. 3.6 and 4.7 dual-use the same 113-video stratified holdout for early stopping and for AP weight estimation when training on all labelled data. The authors correctly flag that holdout member scores are mildly optimistic, yet the final ensemble weights that produce the locked private-test predictions still depend on that dual use. A nested or repeated holdout (or leave-one-participant-out AP) would quantify how much the reported 0.731-class public-test figure and the private submission depend on this coupling; the current design leaves residual contamination risk on the all-data track.
minor comments (6)
  1. Fig. 1 caption states that gaze/brow, prosody, and cue supervision are excluded because they hurt the model; Sec. 4.8 quantifies this only for gaze/brow (AP 0.860→0.841). A one-line table or parenthetical for prosody and cue-supervision drops would make the exclusion claim fully checkable.
  2. Eq. (2) writes BCEw(p,y)=-,w,ylogp-(1-y)log(1-p); the leading comma after the minus sign is a typesetting error and the positive-class weight w is applied only to the positive term, which should be stated explicitly.
  3. Table 7 reports AMF holdout F1 0.751 below several encoder members, yet the public-test single-member AMF is 0.725 (Table 3) and is kept because “the leak-free public test favours it.” A short note reconciling holdout vs public-test ranking of AMF would reduce reader confusion.
  4. The abstract and Sec. 1 state “sixteen features” for ASR-erased time and “eleven” hesitation markers; Sec. 3.3 later refers to a “25-dim marker pathway.” Clarify whether 25 = 11 + 14 (FER stats) or another concatenation so dimensions stay consistent.
  5. Related Work cites the previous winner as 0.694 public / 0.715 private [2]; the private number is not used elsewhere. Either drop it or note that private scores are unavailable for the present submission so the comparison is public-only.
  6. Minor prose: “ABA W” with a space appears once in Sec. 2; “co-attention baseline 0.59” should be 0.590 for consistency with Table 3.

Circularity Check

0 steps flagged

No circularity: empirical ablations and held-out test comparisons; AP weights and fixed threshold are not forced by construction to the reported public-test gains.

full rationale

This is a controlled empirical systems paper on BAH video classification. Its three organising claims (conflict operators do not reliably help; language dominates; AP-weighted averaging at fixed τ=0.5 outperforms validation-tuned weights+threshold) are established by direct measurement on the untouched public test (525 videos) or by ablations that hold encoders/training fixed and vary only one factor (Tables 4–6). Ensemble weights are set by holdout AP (Eq. 3) and the threshold is deliberately fixed at 0.5 precisely to avoid the overfitting the authors themselves document (0.741 val → 0.690 test). The load-bearing 0.731 figure is therefore not a fitted quantity renamed as a prediction; it is an out-of-sample score. Dual use of the 113-video holdout for early stopping and weighting is flagged by the authors as producing mildly optimistic member numbers, but that does not make the public-test comparison circular. No self-definitional equations, no uniqueness theorems imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result appear. Citations are to external encoders, datasets and prior ABAW baselines. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

The central claims rest on standard frozen affect encoders, a small set of hand-chosen hyperparameters, the BAH participant-wise split and labels, and two paper-introduced constructs (AMF gating and ASR-erased time features). No new physical entities; free parameters are ordinary ML knobs, several of which the authors deliberately freeze (τ=0.5, ε=0.1) rather than search on validation.

free parameters (7)
  • marker-head auxiliary loss weight = 0.3
    Set to 0.3 so the marker pathway stays predictive without dominating the main BCE; not grid-searched on test but chosen by hand (Sec. 3.3).
  • label smoothing epsilon = 0.1
    Fixed at the standard 0.1 rather than tuned; used because of small noisy labels (Sec. 3.3).
  • AMF shared projection width = 256
    Architectural width of gated streams set to 256 (Sec. 4.2).
  • decision threshold tau = 0.5
    Fixed at 0.5 for the recommended recipe; deliberately not fitted on validation after the overfitting study (Sec. 3.5, 4.6).
  • ensemble member AP weights = normalized holdout APs
    w_m = AP_m / sum AP_m' computed on validation or the 113-video holdout; threshold-free but still estimated from limited labelled data (Eq. 3).
  • class-balance positive weight w = inverse positive frequency
    w = (N - N+)/N+ from training class counts; standard reweighting, data-dependent (Sec. 3.3).
  • learning rates (text vs other heads) = 2e-5 / 1e-4
    2e-5 for text head with R-Drop; 1e-4 for video/audio/fusion heads (Sec. 4.2).
axioms (5)
  • domain assumption Frozen affect-specialised encoders (RoBERTa-GoEmotions, FER-ViT, emotion wav2vec2, plus VideoMAE/HuBERT members) plus lightweight trained heads are sufficient to expose the BAH signal.
    All main results freeze backbones and train only heads/fusion (Table 1, Sec. 3.2).
  • domain assumption BAH participant-wise splits yield independent train/val/public-test evaluations with reliable binary A/H labels.
    All reported metrics and the claim of beating 0.694 rest on these splits and labels (Table 2, Sec. 4.1).
  • ad hoc to paper Average precision on a small holdout is a stable, threshold-free ranking signal for ensemble weights that transfer better than F1-tuned weights and thresholds.
    Core of the calibration claim and Eq. 3; supported by Table 6 but still an operational assumption on ~100–124 videos.
  • domain assumption Whisper (or equivalent) chunk timestamps remain monotone within 30 s segments so gap features are well-defined after filler deletion.
    ASR-erased time construction explicitly counts gaps only where the timeline is monotone (Sec. 3.4).
  • standard math Standard binary cross-entropy with class reweighting and label smoothing is an adequate training objective for noisy video-level A/H labels.
    Loss in Eq. 2; no custom theoretical derivation required.
invented entities (2)
  • Affective Marker Fusion (AMF) no independent evidence
    purpose: Reliability-gated fusion of text, video, audio, and 11 hesitation markers with an auxiliary marker-only head.
    Named fusion module (Eq. 1–2); standard gating architecture, not an external physical entity. Independent evidence is only the BAH ablations in this paper.
  • ASR-erased time (16-d feature set) no independent evidence
    purpose: Recover hesitation timing that ASR deletes from text by measuring inter-chunk gaps, speaking-rate variability, pre-contrast pauses, and restarts.
    New feature construction from existing timestamps; falsifiable on other ASR+timestamp corpora, but only validated here on BAH (AP 0.718).

reviewed 2026-07-14 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video." pith.science (2026). https://pith.science/paper/IC6SL67D

@misc{pith2026260711120,
  author       = {Pith},
  title        = {Pith review of: Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IC6SL67D}},
  note         = {Machine review of arXiv:2607.11120}
}
Share X Bluesky LinkedIn Reddit HN
abstract

We address ambivalence and hesitancy (A/H) recognition in the ABAW 2026 BAH Challenge: given a short interview video, predict whether the person shows signs of A/H. Our system combines affect-specialised text, audio, and visual representations with a small set of readable linguistic hesitation cues, fused by a reliability gate we call Affective Marker Fusion (AMF), and finished with a simple AP-weighted ensemble at a fixed decision threshold. We also introduce \emph{ASR-erased time}: speech recognisers delete fillers and hesitation pauses from the transcript, but the chunk timestamps keep the time those events took, and sixteen features built from these gaps form the strongest and most independent non-verbal channel we measured (AP $0.718$, correlation $0.11$--$0.36$ with all other members). Across controlled experiments we find three things: cross-modal conflict design does not reliably help on BAH; language is by far the strongest channel while affect-specialised audio is a useful second; and calibration matters more than architecture. Fitting ensemble weights and a threshold on the small validation split overfits: it scores $0.741$ macro-F1 on validation but only $0.690$ on the untouched test set. AP-weighting at a fixed threshold instead reaches $\mathbf{0.731}$ on test.

Figures

Figures reproduced from arXiv: 2607.11120 by Aditya Mishra, Haroon R. Lone, Vikas Kumar.

Figure 1
Figure 1. Figure 1: Overview. Text, video, and audio inputs are turned into affect-specialised features. AMF gates weak channels and predicts A/H. Six [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 7 linked inside Pith

  1. [1]

    Bah dataset for am- bivalence/hesitancy recognition in videos for behavioural change,

    M. Gonz ´alez-Gonz´alez, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger, “Bah dataset for am- bivalence/hesitancy recognition in videos for behavioural change,”arXiv preprint arXiv:2505.19328, vol. 3, no. 9, 2025

  2. [2]

    Conflict-aware multi- modal fusion for ambivalence and hesitancy recognition,

    S. E. Bekhouche, H. Telli, A. Benlamoudi, S. E. Herrouz, A. Taleb-Ahmed, and A. Hadid, “Conflict-aware multi- modal fusion for ambivalence and hesitancy recognition,” arXiv preprint arXiv:2603.15818, 2026

  3. [3]

    Hsemotion team at abaw-8 com- petition: Audiovisual ambivalence/hesitancy, emotional mimicry intensity and facial expression recognition,

    A. V . Savchenko, “Hsemotion team at abaw-8 com- petition: Audiovisual ambivalence/hesitancy, emotional mimicry intensity and facial expression recognition,” arXiv preprint arXiv:2503.10399, 2025

  4. [4]

    The 6th affec- tive behavior analysis in-the-wild (abaw) competition,

    D. Kollias, P. Tzirakis, A. Cowen, S. Zafeiriou, I. Kotsia, A. Baird, C. Gagne, C. Shao, and G. Hu, “The 6th affec- tive behavior analysis in-the-wild (abaw) competition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4587–4598, 2024

  5. [5]

    Context-dependent sen- timent analysis in user-generated videos,

    S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P. Morency, “Context-dependent sen- timent analysis in user-generated videos,” inProceedings of the 55th annual meeting of the association for com- putational linguistics (volume 1: Long papers), pp. 873– 883, 2017

  6. [6]

    Verbal and nonverbal clues for real-life deception detection,

    V . P´erez-Rosas, M. Abouelenien, R. Mihalcea, Y . Xiao, C. Linton, and M. Burzo, “Verbal and nonverbal clues for real-life deception detection,” inProceedings of the 2015 conference on empirical methods in natural lan- guage processing, pp. 2336–2346, 2015

  7. [7]

    Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training,”Advances in neural infor- mation processing systems, vol. 35, pp. 10078–10093, 2022

  8. [8]

    Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units,”IEEE/ACM transactions on au- dio, speech, and language processing, vol. 29, pp. 3451– 3460, 2021

  9. [9]

    Dawn of the transformer era in speech emotion recog- nition: closing the valence gap,

    J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recog- nition: closing the valence gap,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 10745–10759, 2023

  10. [10]

    Roberta: A robustly optimized bert pretraining ap- proach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining ap- proach,”arXiv preprint arXiv:1907.11692, 2019

  11. [11]

    Goemotions: A dataset of fine-grained emotions,

    D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “Goemotions: A dataset of fine-grained emotions,” inProceedings of the 58th annual meeting of the association for computational linguistics, pp. 4040–4054, 2020

  12. [12]

    Deberta: Decoding-enhanced bert with disentangled attention,

    P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv preprint arXiv:2006.03654, 2020

  13. [13]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInterna- tional Conference on Machine Learning, pp. 1321–1330, PMLR, 2017

  14. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly,et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020

  15. [15]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,”Advances in neural informa- tion processing systems, vol. 33, pp. 12449–12460, 2020

  16. [16]

    Rethinking the inception architecture for computer vision,

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wo- jna, “Rethinking the inception architecture for computer vision,” inProceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2818–2826, 2016

  17. [17]

    Robust speech recog- nition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recog- nition via large-scale weak supervision,” inInterna- tional conference on machine learning, pp. 28492–28518, PMLR, 2023

  18. [18]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,”arXiv preprint arXiv:1711.05101, 2017. 5

  19. [19]

    R-drop: Regularized dropout for neural networks,

    L. Wu, J. Li, Y . Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, T.-Y . Liu,et al., “R-drop: Regularized dropout for neural networks,”Advances in neural information processing systems, vol. 34, pp. 10890–10905, 2021

  20. [20]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

  21. [21]

    Transformers: State-of-the-art natural language process- ing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz,et al., “Transformers: State-of-the-art natural language process- ing,” inProceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, 2020

  22. [22]

    Efron and R

    B. Efron and R. J. Tibshirani,An introduction to the boot- strap. Chapman and Hall/CRC, 1994

  23. [23]

    Mediapipe: A frame- work for building perception pipelines,

    C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee,et al., “Mediapipe: A frame- work for building perception pipelines,”arXiv preprint arXiv:1906.08172, 2019. 6

This paper was first reviewed by grok-4.5 on July 14, 2026.