CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

Mingjian Liang; Pamela Carreno-Medrano; Richard Attfield; Wenzhuo Sun; Xuelian Cheng; Zongyuan Ge

REVIEW 4 major objections 5 minor 11 references

A three-expert unanimity gate lets a multimodal video system recognise ambivalence and hesitancy with 0.7771 Macro-F1 on the ABAW11 public test set.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:25 UTC pith:KTECATOB

load-bearing objection Honest challenge report with a useful ensemble rule, but the headline RG-MEC gain is not an independent test result—still worth referee time. the 4 major comments →

arxiv 2607.29310 v1 pith:KTECATOB submitted 2026-07-31 cs.CV

CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

Wenzhuo Sun , Mingjian Liang , Richard Attfield , Zongyuan Ge , Xuelian Cheng , Pamela Carreno-Medrano This is my paper

classification cs.CV

keywords ambivalence and hesitancy recognitionmultimodal ensembledecision-level fusionunanimity-gated overrideMacro-F1participant-disjoint evaluationvideo-level affect recognitionfoundation-model features

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that ambivalence and hesitancy — subtle, often contradictory behavioural states — can be recognised at the video level by combining language, voice, facial, and behaviour-derived statistics, and that the combination can be made more reliable with an asymmetric decision rule. It reports a calibrated multimodal committee, CALM-AH, which reaches 0.7525 Macro-F1 (the unweighted average of F1 scores for the positive and negative classes) on the participant-disjoint ABAW11 public test set. The paper's central claim is that adding Reliability-Gated Multi-Expert Consensus (RG-MEC) raises this to 0.7771 by letting a default anchor prediction be overridden only when three complementary experts unanimously agree on the opposite class. A sympathetic reader would care because the rule offers a way to blend powerful but noisy foundation-model signals with a task-specific system without letting any single expert flip a decision.

Core claim

On the paper's own terms, the discovery is that a fixed decision-level ensemble — not additional training — improves A/H recognition by about 2.5 Macro-F1 points. CALM-AH builds 15 modality-subset classifiers from textual, acoustic, visual, and behavioural-statistical features, selects one per subset by validation cross-entropy, calibrates its threshold for validation Macro-F1, and fuses the binary votes with fixed hard-voting weights. RG-MEC then treats that committee as one of four decision sources: the initial system provides the anchor, while CALM-AH, an affective expert, and a language-model semantic verifier form a correction committee. The final label is the anchor's label unless all

What carries the argument

The load-bearing mechanism is RG-MEC, an asymmetric Boolean decision rule with non-exchangeable roles: the final label is the anchor's label unless all three correction experts unanimously predict the opposite class. In logical form, an override to A/H requires not-anchor and all three experts positive; an override to No-A/H requires anchor positive and all three experts negative. This unanimity gate lets the committee correct errors in both directions while ensuring that one or two experts can never change the default. Beneath it, CALM-AH's hard-voting fusion — 15 calibrated binary votes, one per modality combination, combined with fixed weights — supplies the task-specific evidence that an

Load-bearing premise

The load-bearing assumption is that the 0.7771 public-test number came from a rule not fitted to test feedback—yet the paper itself states in §3.3 that RG-MEC's structure was developed during the challenge under the multi-trial submission protocol and informed by earlier submissions, so the gain is trustworthy only if freezing the rule after that feedback counts as independent evaluation.

What would settle it

Run the locked RG-MEC rule on a fresh participant-disjoint split of the challenge data with no design feedback; if the 0.0246 Macro-F1 margin over the anchor does not reproduce, the public-test gain is a post-hoc fitting artefact.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

A multimodal committee over text, audio, face, and behaviour statistics reaches 0.7525 Macro-F1 on the ABAW11 public test set.
Adding the unanimity-gated decision rule yields 0.7771 Macro-F1, a gain of 0.0246 over the committee alone.
The override is bidirectional: a default positive can become negative and a default negative can become positive, but only on unanimous expert agreement.
Isolated expert agreement (one or two of the three correction experts) never changes the prediction, bounding the damage from individual expert errors.
Because the committee alone already beats the anchor, the reported gains come from both the calibrated committee and the gating rule.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the reported gain survives a clean held-out evaluation, RG-MEC's unanimity gate becomes a reusable template for combining a task-specific system with pre-trained experts in other low-sample behavioural-recognition tasks.
A fair comparison would require freezing the RG-MEC rule before any test feedback; the paper's own protocol disclosure suggests the 0.7771 figure should be read as a competition-era heuristic rather than a pure out-of-sample estimate.
A testable extension is to tune the number of experts or the unanimity threshold (for instance, requiring expert agreement plus anchor agreement) and measure where stability and correction rate trade off on the challenge data.
Because the three correction experts cover task-specific, affective, and semantic evidence, the mechanism implies that expert diversity, not model count, carries much of the value; three similar classifiers would likely not show the same effect.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Honest challenge report with a useful ensemble rule, but the headline RG-MEC gain is not an independent test result—still worth referee time.

read the letter

First, the RG-MEC headline number. The paper is upfront in §3.3 that the rule was shaped by earlier challenge submissions on the same public-test split. That means the 0.0246 MF1 gain from 0.7525 to 0.7771 is not an independent out-of-sample estimate. I don't think this is a fatal flaw—the authors label it as a post-hoc, challenge-specific heuristic and separate it from the locked CALM-AH result—but reviewers and readers should not treat the improvement as a generalizable finding.

What's actually worthwhile: CALM-AH is a careful reconstruction of BROTHER for the ABAW11 splits, and that part is solid. The 15 modality combinations, 45 classifiers, validation-based BCE selection, and threshold calibration are described clearly. The decision to keep the released BROTHER weights instead of re-fitting on validation is reasonable regularization. The RG-MEC rule itself is a small but real idea: an anchor-preserving, unanimity-gated committee with non-exchangeable roles. The boolean formulation is clear, and the ablation in Table 2 shows a plausible monotonic progression.

The soft spots beyond the evaluation issue: no code, no error bars, and no direct comparison to BROTHER, even though BROTHER is the main external backboard. The novelty is modest; there's no theoretical reason given for why unanimity gating should generalize, so the contribution is empirical. That's fine for a challenge report.

The paper is coherent, honest, and the authors are not hiding the main weakness. For someone working on ambivalence/hesitancy recognition or decision-level fusion, this is worth reading as an example of a well-documented challenge system with an explicit limitation. I'd like to see a clean validation of RG-MEC on a held-out split before citing the headline number, but the method itself is interesting enough to send to peer review.

Referee Report

4 major / 5 minor

Summary. The paper presents CALM-AH, a multimodal ensemble for video-level ambivalence/hesitancy recognition on the ABAW11 BAH dataset. Four feature branches (text, audio, visual, behavioural statistics) are used to construct 15 modality subsets; for each subset, one of three classifier families is selected by validation binary cross-entropy, and its decision threshold is calibrated for validation Macro-F1. The binary outputs are fused by hard voting with weights taken from BROTHER. The authors then propose RG-MEC, an asymmetric decision-level ensemble in which an initial anchor prediction is overridden only when three correction experts (CALM-AH, AffectGPT, and a GPT-based semantic verifier) unanimously agree on the opposite label. The paper reports CALM-AH at Macro-F1 0.7525 and RG-MEC at 0.7771 on the public test set, and claims the latter as the main contribution.

Significance. The two-stage architecture is clearly formalized in Eqs. (2) to (8), and the paper is commendably transparent about the fact that the RG-MEC structure was developed during the challenge phase under a multi-trial submission protocol. If the RG-MEC improvement were shown to be a genuine out-of-sample gain, the result would be a useful case study in decision-level fusion with non-exchangeable experts. The paper also reports class-wise F1, which is appropriate for the Macro-F1 target. However, the central evidence for RG-MEC is not an independent evaluation, a key component of the rule (the initial anchor) is not defined, and no uncertainty quantification or direct BROTHER baseline is provided. As presented, the significance of the headline claim is substantially weaker than the abstract and conclusion suggest.

major comments (4)

[§3.3, §5.2, Table 2] The headline claim that RG-MEC improves Macro-F1 from 0.7525 to 0.7771 is not an independent test evaluation. The paper states that 'the structure of RG-MEC was developed during the challenge phase, under the organiser-permitted multi-trial submission protocol' and that its design 'was informed by observations from earlier challenge submissions.' The public-test partition is the same partition used to generate that feedback; freezing the rule after development does not remove the contamination. The sentence in §5.1 that public-test comparisons are 'robustness analyses rather than independent model-selection experiments' is therefore in tension with the unqualified claim in the abstract. The authors should either provide a clean evaluation on data never used during development (e.g., nested participant-disjoint cross-validation using only train/validation labels) or explicitly re-label th
[§4.3, Table 2] The 'initial anchor' y0 is never defined. RG-MEC's final prediction depends directly on y0, and Table 2 reports it as a separate configuration, but the manuscript does not say what model produces y0, how it is trained, or which features it uses. Without this definition, the RG-MEC rule is not reproducible and the ablation in Table 2 cannot be interpreted. The same applies to the correction experts: AffectGPT and the GPT-based semantic verifier are named, but no model version, prompting procedure, output decoding, or decision threshold is specified in the main text. These are load-bearing components of the proposed system and must be described.
[§5.2, §4.2] The reported difference between CALM-AH and RG-MEC is small (0.0246 Macro-F1) on a public test set of 525 videos, yet no error bars, bootstrap intervals, or participant-level variability are provided. Without statistical uncertainty, this difference cannot be distinguished from noise. The paper also claims to 'reconstruct' BROTHER but never reports a BROTHER baseline on the same test partition, making it impossible to assess whether the transferred hard-voting weights are beneficial. A comparison to BROTHER (or to re-optimising the weights on the current validation partition) is needed to support the design choice.
[§5.1, §3.3] The provenance of the BROTHER fusion weights w_c is not stated. If these weights were originally tuned on any ABAW11 labelled split (including the public test), their use in CALM-AH would introduce additional contamination beyond the RG-MEC issue. The paper should clarify where the released weights come from and whether they were derived from the same participant-disjoint partitions.

minor comments (5)

[Title, Abstract] The title uses 'ABA W11' instead of 'ABAW11'; please correct the spacing both in the title and in the running header.
[§3.3] The protocol description would benefit from stating explicitly that the private-test set is never used for any development or selection decision, including the design of RG-MEC. Currently the text says private-test feedback is not used, but the public-test feedback is acknowledged; this distinction is crucial and should be highlighted in the abstract.
[§5.2] Table 2 should indicate whether MF1 is the official Macro-F1 defined in Eq. (1); the abbreviation 'MF1' is introduced only later. Adding standard deviations or confidence intervals would improve the table.
[§4.1] The sentence 'Model selection and threshold calibration are therefore decoupled' is clear, but the paper should specify how the final decision threshold is optimized when the validation set is small (124 videos), since threshold selection on a small set can itself introduce variance.
[References] Reference [2] uses '3nd edition'; should be '3rd edition'. In addition, the challenge URL may not be stable after the workshop; consider adding an archival reference.

Circularity Check

0 steps flagged

No significant circularity: the public-test MF1 gain is an empirical outcome; the acknowledged test-set-informed development of RG-MEC is a validity concern, not a circular construction.

full rationale

The derivation chain is not circular by construction. CALM-AH fits feature transformations and classifiers on the training partition, selects models and calibrates thresholds on the validation partition, and then evaluates on the participant-disjoint public test; the released BROTHER weights are external, not self-cited, and are explicitly not re-optimized. RG-MEC is a Boolean decision rule (Eqs. 4-8) that combines four independent expert predictions, and the reported 0.0246 Macro-F1 improvement over standalone CALM-AH is an empirical result, not an identity or a fitted parameter renamed as a prediction. The paper itself flags the main caveat in §3.3: 'The structure of RG-MEC was developed during the challenge phase, under the organiser-permitted multi-trial submission protocol' and its design 'was informed by observations from earlier challenge submissions.' This means the 0.7771 public-test number may be optimistically biased by test-set-informed selection, which is a data-snooping / validity concern rather than circularity. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via the authors' own prior work.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central method rests on a small number of external design choices: transferred BROTHER weights, validation-calibrated thresholds, and a challenge-feedback-derived override rule. No new entities are introduced.

free parameters (5)

BROTHER fusion weights w_c = released weights from BROTHER [4]
Used in Eq. (3); not re-estimated on current data, assumed transferable.
Decision threshold τ_c per modality subset = 15 thresholds, values not listed
Calibrated to maximize validation Macro-F1 (§4.1).
Classifier family per modality subset = 15 selected models (MLP/RF/GBDT)
Selected by validation BCE (§5.1).
Visual PCA dimensionality = 512
PCA reduction to 512 dimensions after standardization (§5.1).
RG-MEC override rule = unanimity (3/3 agreement); developed in challenge phase
Informed by earlier official trial behavior (§3.3), not a validation-only selection.

axioms (6)

domain assumption The BAH expert annotations of A/H are correct ground truth.
Used to train/evaluate; no annotation-quality analysis.
domain assumption Participant-disjoint splits are sufficient to measure generalization to unseen individuals.
Protocol from organizers [1,2]; paper relies on it but does not verify.
ad hoc to paper BROTHER's released hard-voting weights transfer to the reconstructed committee.
Eq. (3) uses fixed weights w_c from BROTHER [4] without re-optimising; no evidence given that these weights suit the ABAW11-rebuilt feature sets.
domain assumption F2LLM, HuBERT, SigLIP2 embeddings and the 102-dim behavioural statistics contain sufficient signal.
No feature-level analysis is provided.
domain assumption AffectGPT and the GPT-based semantic verifier produce usable binary A/H decisions.
Their outputs are gated by unanimity, but no individual performance is reported.
ad hoc to paper Unanimous agreement among the three correction experts is a reliable override signal.
Eq. (8) encodes this; rule was selected post-hoc from challenge submissions, so its reliability is not independently established.

pith-pipeline@v1.3.0-daily-deepseek · 6486 in / 8875 out tokens · 88730 ms · 2026-08-03T09:25:36.495073+00:00 · methodology

0 comments

read the original abstract

Ambivalence and hesitancy (A/H) are subtle behavioural states that may be expressed through language, voice, facial activity, and other non-verbal cues. The ABAW11 A/H Video Recognition Challenge asks systems to assign a binary A/H label to each naturalistic interview video. Performance is measured using Macro-F1 so that recognition of both A/H and No-A/H samples receives equal importance. We present CALM-AH, a multimodal ensemble that combines textual, acoustic, visual, and derived behavioural-statistical features. We construct 15 non-empty combinations of these feature branches. For each combination, we select the best of three classifier families using validation binary cross-entropy and optimise its decision threshold for validation Macro-F1. The resulting binary decisions are combined using fixed hard-voting weights transferred from BROTHER. We further introduce Reliability-Gated Multi-Expert Consensus(RG-MEC), an anchor-preserving decision-level ensemble that combines an initial prediction with three complementary correction experts: CALM-AH, AffectGPT, and a GPT-based semantic verifier. The initial system provides the default prediction. Its label is overridden only when all three correction experts unanimously support the same alternative class; otherwise, the anchor prediction is retained. This unanimity-gated design limits the influence of isolated expert errors while permitting bidirectional correction when task-specific, multimodal-affective, and semantic-pragmatic evidence are fully consistent. On the participant-disjoint ABAW11 dataset, CALM-AH achieves a Macro-F1 of 0.7525, and the complete RG-MEC system achieves 0.7771.

Figures

Figures reproduced from arXiv: 2607.29310 by Mingjian Liang, Pamela Carreno-Medrano, Richard Attfield, Wenzhuo Sun, Xuelian Cheng, Zongyuan Ge.

**Figure 1.** Figure 1: The pipeline of our method. rule and evaluate it on the public-test set. The private-test feedback is not used to update feature encoders, downstream classifier parameters, decision thresholds, or CALM-AH fusion weights. The final RG-MEC-assisted submission rule is reported separately from the controlled public-test method. Its design was informed by observations from earlier challenge submissions and is t… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 3 linked inside Pith

[1]

Gonz\'alez-Gonz\'alez, S

M. Gonz\'alez-Gonz\'alez, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. BAH dataset for ambivalence/hesitancy recognition in videos for digital behavioural change. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.19328

Pith/arXiv arXiv 2026
[2]

Announcement of the 3nd edition: Ambivalence/Hesitancy video recognition challenge, ABAW11 at ECCV 2026

ABAW Organizers. Announcement of the 3nd edition: Ambivalence/Hesitancy video recognition challenge, ABAW11 at ECCV 2026. https://affective-behavior-analysis-in-the-wild.github.io/11th, 2026

2026
[3]

S. E. Bekhouche, H. Telli, A. Benlamoudi, S. E. Herrouz, A. Taleb-Ahmed, and A. Hadid. Conflict-aware multimodal fusion for ambivalence and hesitancy recognition. arXiv preprint arXiv:2603.15818, 2026

arXiv 2026
[4]

Pereira, B

A. Pereira, B. Fernandes, and P. Barros. BROTHER: Behavioral recognition optimized through heterogeneous ensemble regularization for ambivalence and hesitancy. arXiv preprint arXiv:2603.14361, 2026

arXiv 2026
[5]

A. V. Savchenko. HSEmotion team at ABAW-8 competition: Audio-visual ambivalence/hesitancy, emotional mimicry intensity and facial expression recognition. arXiv preprint arXiv:2503.10399, 2025

Pith/arXiv arXiv 2025
[6]

Baltru s aitis, C

T. Baltru s aitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423--443, 2019

2019
[7]

W. R. Miller and S. Rollnick. Motivational Interviewing: Helping People Change. Guilford Press, third edition, 2013

2013
[8]

N. E. MacDonald and the SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition, scope and determinants. Vaccine, 33(34):4161--4164, 2015

2015
[9]

Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=

Real-time segmentation of on-line handwritten arabic script , author=. Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=. 2014 , organization=

2014
[10]

Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=

Fast classification of handwritten on-line Arabic characters , author=. Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=. 2014 , organization=

2014
[11]

arXiv preprint arXiv:1804.09028 , year=

Estimate and Replace: A Novel Approach to Integrating Deep Neural Networks with Existing Applications , author=. arXiv preprint arXiv:1804.09028 , year=

Pith/arXiv arXiv

[1] [1]

Gonz\'alez-Gonz\'alez, S

M. Gonz\'alez-Gonz\'alez, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. BAH dataset for ambivalence/hesitancy recognition in videos for digital behavioural change. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.19328

Pith/arXiv arXiv 2026

[2] [2]

Announcement of the 3nd edition: Ambivalence/Hesitancy video recognition challenge, ABAW11 at ECCV 2026

ABAW Organizers. Announcement of the 3nd edition: Ambivalence/Hesitancy video recognition challenge, ABAW11 at ECCV 2026. https://affective-behavior-analysis-in-the-wild.github.io/11th, 2026

2026

[3] [3]

S. E. Bekhouche, H. Telli, A. Benlamoudi, S. E. Herrouz, A. Taleb-Ahmed, and A. Hadid. Conflict-aware multimodal fusion for ambivalence and hesitancy recognition. arXiv preprint arXiv:2603.15818, 2026

arXiv 2026

[4] [4]

Pereira, B

A. Pereira, B. Fernandes, and P. Barros. BROTHER: Behavioral recognition optimized through heterogeneous ensemble regularization for ambivalence and hesitancy. arXiv preprint arXiv:2603.14361, 2026

arXiv 2026

[5] [5]

A. V. Savchenko. HSEmotion team at ABAW-8 competition: Audio-visual ambivalence/hesitancy, emotional mimicry intensity and facial expression recognition. arXiv preprint arXiv:2503.10399, 2025

Pith/arXiv arXiv 2025

[6] [6]

Baltru s aitis, C

T. Baltru s aitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423--443, 2019

2019

[7] [7]

W. R. Miller and S. Rollnick. Motivational Interviewing: Helping People Change. Guilford Press, third edition, 2013

2013

[8] [8]

N. E. MacDonald and the SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition, scope and determinants. Vaccine, 33(34):4161--4164, 2015

2015

[9] [9]

Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=

Real-time segmentation of on-line handwritten arabic script , author=. Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pages=. 2014 , organization=

2014

[10] [10]

Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=

Fast classification of handwritten on-line Arabic characters , author=. Soft Computing and Pattern Recognition (SoCPaR), 2014 6th International Conference of , pages=. 2014 , organization=

2014

[11] [11]

arXiv preprint arXiv:1804.09028 , year=

Estimate and Replace: A Novel Approach to Integrating Deep Neural Networks with Existing Applications , author=. arXiv preprint arXiv:1804.09028 , year=

Pith/arXiv arXiv