{"id":"232757c5-cfe0-48b9-86c0-ed64944be98a","arxiv_id":"2412.08965","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AFFAKT transfers facial expression knowledge to video deception detection with hierarchical optimal transport and a sample-specific reweighting bank, improving F1, ACC, and AUC on the RTL and DOLOS benchmarks.","lead":"A new computer-vision pipeline, AFFAKT, improves video deception detection by transferring knowledge from large facial expression datasets using hierarchical optimal transport. The method reports state-of-the-art accuracy on two deception benchmarks, and its learned correlations match psychological findings that deception is linked to negative emotions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Audio/fused gains may come from learned re-mapping rather than transferred expression knowledge; source-semantics ablation is missing.","rationale":"The reader's weakest assumption is the same cross-modal alignment issue; my read agrees that this is the right place to look, but the more precise threat is that the method's audio/fused gains may not require any facial expression semantics at all. A source-shuffle/random-source ablation would settle this, and until it is run the unconditional version of the SOTA claim is not established. I do not see an internal contradiction that forces rejection: the equations are mostly coherent, the visual-modality results are consistent with the psychological story, and the SRKB module is parameter-free as claimed. Minor additional concerns — unreported epsilon, dataset-specific xi and nu, and the row-sum drift in the momentum update of B in Eq. (12) — are worth checking but are secondary to the semantic-transfer question. Therefore the reader's CONDITIONAL verdict is appropriate and my pass does not move it.","tokens_in":25358,"tokens_out":5688,"duration_ms":63155,"concrete_test":"Run RTL and DOLOS audio and fused experiments under two source-feature ablations: (1) shuffle the source labels before forming the class centroids in Eq. (6), keeping all other settings fixed; (2) replace Xs with fixed random vectors of the same dimension. If ACC/F1/AUC stay within the reported fold standard deviations, the semantic content of the expression classes is not the driver and the transfer claim for those modalities fails. Also report the entropic regularization constant epsilon used in Eqs. (3) and (5), since T and hence Xtrans depend on it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that transferring facial expression knowledge via hierarchical optimal transport improves deception detection. The weakest load-bearing point is the cross-modal case: in Eqs. (3)-(6), Xs are visual MAE-DFER embeddings of expression classes, while for the audio modality Xt are W2V2 audio embeddings. A single linear map F1 is expected to make cosine similarity and OT distances between audio target samples and visual expression class centroids semantically meaningful. If F1 and F2 instead learn arbitrary projections, Xtrans in Eq. (6) is a sample-dependent learned bias, not affective knowledge, and the reported audio/fused improvements do not establish the transfer claim. The paper's interpretability section itself acknowledges that the visual/audio modality discrepancy 'hinders the establishment of the relationship.' No experiment replaces or shuffles the source features to verify that the semantic content of the expression classes, rather than the added capacity or the OT weighting mechanism, is responsible for the gains. This is testable and is the main unresolved threat to the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses video deception detection under limited labeled data by transferring knowledge from large video facial expression recognition datasets. It proposes AFFAKT with two modules: H-OTKT, which computes a hierarchical optimal transport plan between target deception samples and source expression classes and uses the plan to weight class-average source features added to target features; and SRKB, which maintains a momentum-updated class-level correlation prototype and re-weights test-time transport plans using a standard-deviation threshold. The method is evaluated on RTL and DOLOS for visual, audio, and fused modalities, using DFEW, FERV39K, and MAFW as source domains, with ablation, sensitivity, and interpretability analyses. The central claim is that transferring facial-expression knowledge through this scheme improves deception detection across all reported settings and outperforms the comparison methods, including PECL, FreeLunch, ADC, and Cr-KD-NCD.","tokens_in":25576,"tokens_out":7041,"duration_ms":75115,"significance":"If the empirical claim holds, the paper offers a useful and reasonably interpretable way to exploit large affective datasets for a small-sample task, and the reported fold-level standard deviations and ablations are commendable. The modular description is clear, the code is promised, and the learned correlation prototypes broadly align with prior psychological findings, which adds credibility to the visual-modality results. The main unresolved issue is whether the audio and fused improvements are due to semantic expression knowledge or to learned re-mapping and added capacity; this needs a source-content ablation before the central transfer claim can be fully accepted. The missing value of the entropic regularization parameter and the absence of a stated model-selection protocol also weaken reproducibility and the strength of the empirical comparison.","major_comments":[{"comment":"The entropic regularization coefficient ε appears in the Sinkhorn problems in Eqs. (2), (3), and (5) and directly determines the low-level and high-level transport plans T_low,k and T, which in turn determine Xtrans and the correlation prototype B in Eq. (12). However, ε is never reported in the main text or the appendix. Without this value, including any schedule or per-source/per-modality settings, the results in Tables 1 and 5 are not reproducible, and the sensitivity of the method to this central parameter is unknown. Please report ε for all source–target–modality combinations and add a sensitivity analysis for it.","section":"Eqs. (2), (3), (5) and Experimental Settings"},{"comment":"The load-bearing claim that transferred features carry facial-expression semantics is not established for the audio and fused modalities. In Eqs. (3)–(6), Xs are visual MAE-DFER embeddings, while for the audio experiments Xt are W2V2 audio embeddings; a single linear layer F1 is expected to align these heterogeneous spaces before cosine-based OT costs are computed, and Xtrans in Eq. (6) can act as a learned sample-dependent bias if F2 absorbs the mismatch. The appendix itself states that the visual/audio modality discrepancy 'hinders the establishment of the relationship,' and the ablation in Table 2 shows that H-OTKT alone decreases accuracy on several audio and fused conditions. No experiment replaces or shuffles the source features to verify that the semantic content of the expression classes, rather than the added capacity of F2 or the OT reweighting mechanism, is responsible for the gains in Table 5(b,c). Please add a source-semantics ablation, for example random source features, permuted expression labels, or a matched audio emotion source, and report whether the audio and fused improvements persist.","section":"Eqs. (3)–(7), Table 5(b,c), and appendix Interpretability Studies"},{"comment":"The hyperparameters ξ, ν, η, and α are set differently across datasets and modalities (for example, ξ=0.5 for RLT audio but 0.2 for visual and fused, and ν=0.05 versus 0.1 in different conditions), but the paper does not describe the model-selection protocol, and the sensitivity analysis in Fig. 5 is run only for RLT under the visual modality. Consequently, the reader cannot determine whether the reported improvements come from the method itself or from per-dataset/per-modality tuning on the test folds. Please specify how each hyperparameter was chosen, ideally through a validation split within each cross-validation fold, and provide sensitivity results for the audio and fused settings and for DOLOS, or demonstrate that the fixed settings transfer across conditions.","section":"Experimental Settings and Sensitive Analyses"}],"minor_comments":[{"comment":"The dataset name is written inconsistently as RLT in some tables and RTL in the text; please standardize the abbreviation.","section":"Throughout"},{"comment":"The curriculum-learning schedule for ξ′ is ambiguous as printed: it appears as ξ′ = ξ 2 × (1 − cos(e−1 Ne × π)) with unclear fraction and exponent. Please rewrite the formula with explicit parentheses and define e.","section":"After Eq. (7)"},{"comment":"There are typographical errors such as 'resutls', 'donated' for 'denoted', and 'Tranfer' in Figure 3; these should be corrected.","section":"Experiments, Ablation Studies"},{"comment":"The standard deviations are reported across folds only, with no random seeds or repeated runs; a sentence clarifying that the reported variance is fold-to-fold and not run-to-run would prevent over-interpretation of the ± values.","section":"Table 5"},{"comment":"The logistic-regression importance weights p_j^k in the low-level OT are mentioned but never described; please state what features and labels are used and whether the logistic regressors are trained on the source domain or reused from prior work.","section":"Appendix, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The main risk for the paper is that the audio/fused results are perceived as an artifact of added model capacity rather than of semantic knowledge transfer; asking for a source-feature ablation is essential. Also verify that the public repository actually contains the code, the exact hyperparameters including ε, and the random seeds used for the five folds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on AFFAKT.\n\nThe novel bit is real: applying hierarchical optimal transport from Guo et al. 2022 to transfer facial expression knowledge to video deception detection, plus the SRKB sample-specific reweighting during inference. The paper is clearly written, the ablations are sensible, and the RTL visual gains are large (F1 0.876 vs 0.779 for the best OT baseline). The interpretability table showing sad/fear loading on deceptive and happy on truthful matches the psychology literature. That deserves credit.\n\nThe soft spots, in order of importance. First, the audio and fused modality transfers are the weakest link. The source features are always visual MAE-DFER embeddings, the target audio features are W2V2 embeddings, and a single linear MLP F1 is supposed to make cosine distances between them semantically meaningful. The paper itself concedes in the interpretability section that the modality discrepancy 'hinders the establishment of the relationship.' No experiment replaces or shuffles the source expression class features to show that the semantic content, rather than the added parameter capacity or the OT weighting mechanism, is what drives the gains. That is a testable ablation and it is missing. Until that is done, the mechanism claim should be read as suggestive, not established.\n\nSecond, several small reporting gaps: the entropic regularization epsilon is never given, and xi, nu, eta, alpha are tuned per dataset and per modality. That is not fatal; the sensitivity analysis covers four of them, but it does weaken the claim that the approach is robust rather than well-tuned. Third, on the larger DOLOS set the improvements over the ADC baseline are modest: e.g., visual F1 +0.017, ACC +0.005. The paper's own text says gains are more significant on the smaller RTL, which is consistent with the method's purpose but worth stating plainly.\n\nThe citation and related work look fine. The code is promised on GitHub, which is good. I don't see a circularity problem: using target labels to update the prototype B is standard supervised learning.\n\nBottom line: this is a solid applied paper with a plausible empirical claim and a real gap in the causal evidence for its mechanism. A serious referee should engage with it, but the revision must include a source-semantics ablation (e.g., shuffled or replaced source class features) before I'd trust the 'affective knowledge transfer' interpretation. I'd send it to peer review.","headline":"AFFAKT is a genuine OT-transfer recipe for deception detection with strong RTL gains, but the audio/fused improvements and the 'affective knowledge' mechanism need an ablation that removes or shuffles source semantics before the claim is fully trustworthy.","tokens_in":26073,"tokens_out":2226,"would_cite":false,"duration_ms":22385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AFFAKT transfers facial-expression knowledge to improve video deception detection under scarce labels.","keywords":["video deception detection","optimal transport","knowledge transfer","facial expression recognition","affective computing","hierarchical optimal transport","multimodal deception detection","low-resource learning"],"falsifier":"Retrain AFFAKT after randomly permuting the facial-expression class labels attached to the source features, or after replacing source features with noise of the same dimension; if accuracy on RTL or DOLOS stays close to the reported values, the performance gain comes from the optimal transport machinery rather than from expression semantics.","tokens_in":25151,"feed_emoji":"🎭","tokens_out":6719,"duration_ms":66019,"temperature":0.7,"pith_summary":"Video deception detection suffers from a shortage of large labeled datasets. To overcome this, AFFAKT borrows knowledge from large facial expression recognition datasets, reasoning that expressions carry cues to deception. The paper's central claim is that a hierarchical optimal transport plan between facial expression classes and deception samples can decide how much knowledge each expression category should contribute to each target sample, and that a momentum-updated correlation prototype can stabilize this mapping at test time. On the RTL and DOLOS deception benchmarks, the method reports the best F1, accuracy, and AUC among the compared methods across visual, audio, and fused modalities. A learned prototype shows that deceptive samples align more strongly with sadness and fear, while truthful samples align with happiness, matching psychological findings.","feed_headline":"Facial-expression transfer tops deception-detection benchmarks","feed_subtitle":"A hierarchical optimal-transport scheme decides how much of each expression to borrow, lifting RTL visual F1 to 0.876.","key_machinery":"The central object is a two-level, entropy-regularized optimal transport scheme. Low-level OT solves, for each source expression class, a transport plan between target samples and individual source samples using a cosine cost; these per-class OT distances form the cost matrix for high-level OT, whose solution T is the correlation mapping between expression classes and deception samples. Equation (6) turns T into transferred features by weighting class-mean source features, and equation (7) fuses them with target features under a curriculum-controlled weight. The SRKB module adds a momentum-updated prototype B, one row per target class, and a sample-specific reweighting rule that replaces or blends a noisy transport plan with the prototype during testing. Together these components carry the argument that the amount and the use of transferred affective knowledge can both be learned rather than hand-set.","core_discovery":"The paper argues that facial expression data, despite having a different label space and feature distribution, can be a useful auxiliary source for deception detection when the amount of labeled deception data is small. The core procedure is H-OTKT: low-level optimal transport computes a per-class cost between each target sample and individual source samples, and high-level optimal transport uses these costs to solve for a transport plan T that assigns each deception sample a weight over expression classes. Those weights produce transferred features as weighted class centroids, which are fused with the target features through a curriculum-weighted sum. A second module, SRKB, maintains a momentum-updated correlation prototype B for truthful and deceptive classes and, at inference, re-weights the estimated transport plan toward this prototype when the plan's standard deviation is too small to be trusted. On the visual RTL benchmark with DFEW, a large in-the-wild facial expression dataset, as source, AFFAKT reports F1 0.8760, accuracy 0.8670, and AUC 0.8789, and it reports the best average results among the compared methods on both datasets and modalities.","pith_inferences":["An untested extension is whether the same hierarchical transport recipe transfers other affective source domains, such as speech emotion, to other low-resource tasks like depression or pain detection; the paper does not report such experiments.","The paper does not isolate how much of the gain comes from the expression semantics in the source labels versus from the optimal transport geometry; a reader should treat the semantic explanation as a hypothesis until an ablation with shuffled source labels is run.","Since the authors show that the choice of source encoder affects results, a practical consequence we draw is that improving source feature compactness may matter more than increasing source dataset size for this transfer scheme.","The cross-modal audio result uses visual source features mapped by a single linear MLP; in our reading, this is the least protected claim and the most natural stress test for the method's core assumption."],"forward_implications":["If AFFAKT's central claim is correct, a large facial expression corpus can serve as a practical knowledge source for deception detection even when target labels are scarce.","The transfer works not only within the visual modality but also from visual expression features to audio deception features, suggesting cross-modal affective knowledge transfer is feasible.","The learned correlation prototype gives an interpretable decomposition of deceptive and truthful behavior into expression associations, offering cues for psychology-informed model design.","Because the method outperforms its strongest comparisons most clearly on the smaller RTL dataset, the approach is likely most valuable precisely when labeled deception data are hardest to obtain."],"supporting_citations":[{"why":"Supplies the DOLOS dataset, the PECL baseline, and the adapter-tuning and backbone setup that AFFAKT builds on.","marker":"Guo et al. 2023"},{"why":"Provides the RTL real-life trial deception dataset used as the second target benchmark.","marker":"Pérez-Rosas et al. 2015"},{"why":"Introduces the Sinkhorn algorithm that solves the entropy-regularized optimal transport problems in Eqs. (3) and (5).","marker":"Cuturi 2013"},{"why":"Provides the hierarchical optimal transport formulation and per-sample importance weighting that H-OTKT adapts for knowledge transfer.","marker":"Guo et al. 2022"},{"why":"Supplies the psychological evidence linking deception to facial expression cues, motivating the transfer direction and the interpretability analysis.","marker":"DePaulo et al. 2003"},{"why":"Provides the MAE-DFER pretrained encoder used to extract source facial expression features.","marker":"Sun et al. 2023"},{"why":"Provides the VideoMAE backbone used to encode target video features.","marker":"Tong et al. 2022"},{"why":"Provides the W2V2 backbone used to encode target audio features.","marker":"Baevski et al. 2020"},{"why":"Supplies the Sinkhorn divergence used in Eq. (10) to align source and target feature spaces during training.","marker":"Feydy et al. 2019"}],"fun_headline_variants":["Facial expressions: secret weapon for deception detection","Optimal transport maps expressions to expose deception","AFFAKT: borrow expression knowledge, spot lies better","F1 0.876: expression transfer boosts deception spotting","How much expression to transfer? Optimal transport decides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer rests on the assumption that one learned linear projection makes facial-expression and deception-video features similar enough that cosine-based optimal transport between them reflects real affective correspondence, including when the source is visual and the target is audio.","fun_headline_variants_meta":{"raw":{"variants":["Facial expressions: secret weapon for deception detection","Optimal transport maps expressions to expose deception","AFFAKT: borrow expression knowledge, spot lies better","F1 0.876: expression transfer boosts deception spotting","How much expression to transfer? Optimal transport decides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1583,"prompt_tokens":1000,"completion_tokens":583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":507}},"tokens_in":616,"tokens_out":583,"duration_ms":6391,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:24.848004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain AFFAKT after randomly permuting the facial-expression class labels attached to the source features, or after replacing source features with noise of the same dimension; if accuracy on RTL or DOLOS stays close to the reported values, the performance gain comes from the optimal transport machinery rather than from expression semantics.","supporting_citations":[{"cited_title":"M.; Yu, Z.; Kong, A","cited_arxiv_id":null,"evidence_quote":"Supplies the DOLOS dataset, the PECL baseline, and the adapter-tuning and backbone setup that AFFAKT builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RTL real-life trial deception dataset used as the second target benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hierarchical optimal transport formulation and per-sample importance weighting that H-OTKT adapts for knowledge transfer."},{"cited_title":"M.; Lindsay, J","cited_arxiv_id":null,"evidence_quote":"Supplies the psychological evidence linking deception to facial expression cues, motivating the transfer direction and the interpretability analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MAE-DFER pretrained encoder used to extract source facial expression features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Sinkhorn divergence used in Eq. (10) to align source and target feature spaces during training."}],"review_version":1}