{"id":"cd5d736e-b4a1-4833-ba72-2f97fa262ae1","arxiv_id":"2505.14151","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"ReactDiff generates multiple listener facial reactions from a speaker's audio and video using a multi-modality transformer with latent diffusion, but its reported benchmark superiority conflicts with its own tables.","lead":"The paper describes a neural network, ReactDiff, that watches and listens to one person in a conversation and generates several plausible facial reactions for the listener. It combines a transformer that mixes audio and video cues with a diffusion model that adds randomness, then tests the system on the REACT2024 benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SOTA claim is contradicted by the paper's own tables: base ReactDiff is not best on FRCorr, FRDiv, or FRRea, and the LRS variant that wins several metrics is never defined.","rationale":"The reader's verdict of REJECT is well supported by the paper's internal contradictions, and I agree that the central claim is not established. However, the reader's weakest_assumption focuses on the frozen pretrained feature extractors (ELIM, ME-GraphAU, ResMaskNet, wav2vec2). That is a real secondary risk, but it is not the most load-bearing concern: even if those extractors are perfect, the paper's own Table 1 shows PerFRDiff with FRCorr 0.38 versus ReactDiff's 0.26, and Table 2 shows Liu et al. and PerFRDiff with higher FRDiv than ReactDiff's 0.0940. The text repeatedly asserts that ReactDiff has the highest scores when the tables show otherwise. The strongest problem is internal inconsistency between prose and tables, plus the undefined LRS variant that accounts for the best results on several metrics. The reader's rationale does identify the table contradiction and the undefined LRS, so my agreement is partial: the verdict aligns, but the weakest_assumption field does not name the decisive issue. A single targeted check—independently recomputing the main FRCorr numbers and locating an LRS definition in the released code—would settle whether the headline claim is numerically true or whether the paper needs major revision.","tokens_in":18094,"tokens_out":3769,"duration_ms":32268,"concrete_test":"Run the official REACT2024 evaluation code on the released ReactDiff repository using the exact Table 1 setup, and independently compute FRCorr for base ReactDiff, ReactDiff (LRS), and PerFRDiff. If the reproduced numbers match the table, the abstract's claim is contradicted because base ReactDiff's 0.26 is below PerFRDiff's 0.38; if the numbers differ, the table is unreliable. In either case, check the repository for any configuration or code that defines 'LRS'; absence of such a definition confirms that the winning variant's results are non-reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ReactDiff 'significantly outperforms existing approaches' with FRCorr 0.26 and FRDiv 0.094 stands or falls on the experimental tables. Table 1 lists PerFRDiff at FRCorr 0.38, ReactDiff (LRS) at 0.29, and base ReactDiff at 0.26; the text in Section 4.3.1 nevertheless asserts ReactDiff achieves the highest FRCorr of 0.26 among all methods. That is internally inconsistent. Table 2 lists Liu et al. at FRDiv 0.1675 and PerFRDiff at 0.1368, both above ReactDiff's 0.0940, again contradicting the claim of superior diversity. Table 3 lists PerFRDiff at FRRea 47.62, better (lower) than ReactDiff's 66.72, while the text says ReactDiff's 66.72 is the best. The only variant that does win several metrics, ReactDiff (LRS) (FRDist 79.63, FRDvs 0.1380, FRSyn 39.40), is introduced in Section 3.4 with one sentence mentioning 'Listener Reaction Shifts' but no architectural definition, training procedure, or hyperparameters, so its results cannot be reproduced or attributed. Thus the load-bearing condition for the central claim is not primarily the quality of the frozen feature extractors; it is the validity of the comparison evidence itself. Even if the feature extractors are imperfect, the reported numbers as written fail to establish that the proposed architecture outperforms prior work. The paper may have a salvageable core, but as submitted the abstract's claim is not supported by its own experimental section.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReactDiff combines a Multi-Modality Transformer (MMT) with a conditional DDIM in a latent reaction space for facial reaction generation. The pipeline extracts facial (VA, AU, FE) and acoustic (MFCC, wav2vec2) features, encodes them through intra- and inter-class attention, applies a diffusion process conditioned on the speaker's behavior constraint vector, and decodes into 3DMM and emotion coefficients. Experiments are reported on the REACT2024 offline and online challenge subsets, with the paper claiming state-of-the-art performance in appropriateness, diversity, realism, and synchrony.","tokens_in":18459,"tokens_out":5690,"duration_ms":53426,"significance":"If the experimental claims were correct, the combination of hierarchical multi-modal attention with latent-space diffusion would be a useful contribution to the REACT2024 challenge, which is a concrete benchmark with clear evaluation protocols. The authors also make the code open-source, and the training procedure is described in enough detail to be reimplemented. However, the central comparative evidence is internally inconsistent: the paper's own tables show that base ReactDiff is not state of the art on the metrics highlighted in the abstract, and the best reported variant (ReactDiff LRS) is never defined. The claimed significance is therefore not established by the manuscript as submitted.","major_comments":[{"comment":"The text states that 'ReactDiff achieves the highest FRCorr score of 0.26 among all methods,' but Table 1 lists PerFRDiff at FRCorr 0.38, VQ-Diff at 0.30, and Liu et al. at 0.22 in the offline setting, with ReactDiff at 0.26. In the online setting, VQ-Diff at 0.30 is again above ReactDiff's 0.24. The highest-FRCorr claim is directly contradicted by the paper's own comparison table.","section":"§4.3.1, Table 1"},{"comment":"The text claims that 'ReactDiff demonstrates superior performance across all diversity metrics.' Table 2 shows that Liu et al. achieves FRDiv 0.1675 and PerFRDiff achieves FRDiv 0.1368, both above ReactDiff's 0.0940; PerFRDiff achieves FRVar 0.2191 versus ReactDiff's 0.0462; and Liu et al. achieves FRDvs 0.1385 versus ReactDiff's 0.1012. The only variant that reaches some top diversity scores is the undefined ReactDiff (LRS), whose mechanism appears nowhere in the method section.","section":"§4.3.2, Table 2"},{"comment":"For the realism metric FRRea, lower is better. Table 3 lists PerFRDiff at 47.62 and ReactDiff at 66.72, so the sentence 'ReactDiff achieves an FRRea score of 66.72, which is the best among the compared methods' is false. In the online setting, Trans-VAE at 69.19 is better than ReactDiff's 73.20, again contradicting the text that claims best realism.","section":"§4.3.3, Table 3"},{"comment":"ReactDiff (LRS) is introduced in §3.4 with the single sentence that the authors 'introduce Listener Reaction Shifts (LRS) to model personalized cognitive styles,' but no architectural definition, training objective, algorithm, or hyperparameters are provided. This variant is nevertheless the one reported as best in FRDist (79.63), FRDvs (0.1380), and FRSyn (39.40). Because the variant is unspecified, the strongest reported results cannot be reproduced and cannot be attributed to the described architecture.","section":"§3.4, Tables 1–3"},{"comment":"The ablation table does not support the conclusion that the diffusion component helps the headline metrics. Removing diffusion yields FRCorr 0.84 versus 0.26, FRDist 80.10 versus 86.70, and FRDvs 0.1349 versus 0.1012, all better than the full model. The text says removing diffusion 'eliminates diversity,' but FRDvs is higher without diffusion; it also says the full model has 'optimal diversity scores,' which the table does not show. The contribution of the diffusion module to the reported improvements is therefore not established.","section":"§4.4, Table 4"}],"minor_comments":[{"comment":"The abstract's headline numbers (FRCorr 0.26 and diversity 0.094) are presented as superior to prior work, but Tables 1 and 2 list higher values for PerFRDiff and Liu et al.; all comparative claims in the abstract and text should be reconciled with the tables.","section":"Abstract and §4.3.1"},{"comment":"The DDIM sampling schedule at test time is not specified: the paper does not state the value of eta, the number of sampling steps, or whether stochastic noise is added during generation, all of which are important for reproducing the reported diversity and realism results.","section":"§3.4, Eq. (8)"},{"comment":"The AU feature dimension of 25,088 is unusually large and is not motivated; the paper should explain the projection of these features and why ME-GraphAU features without a classifier are used at this raw dimension.","section":"§3.2.1 and Eq. (3)"},{"comment":"The training details report 'training time of two stages is 16h' but do not report inference time or per-epoch cost; these are relevant to the claimed suitability for real-time interaction scenarios.","section":"§4.2"},{"comment":"The citation '(Dosovitskiy, 2020)' is used to support a statement about the evolution of HCI; this reference is about vision transformers and is not appropriate here. An HCI-related reference should be used instead.","section":"§2.2"}],"recommendation":"reject","confidential_remarks":"The ReactDiff (LRS) variant is a significant red flag: it appears in the result tables and in the claims of best performance, but the method section contains only one vague sentence about it. If the authors ever resubmit a revised version, the editor should require either a complete specification of LRS and its training/inference procedure, or the removal of all LRS-based results from the tables and claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the architecture is a reasonable combination—MMT with intra/inter-cross attention plus a latent DDIM conditioned on the speaker—and the two-stage training is concrete enough to reproduce. But the paper's own tables sink the headline claim. Table 1 shows PerFRDiff at FRCorr 0.38 and ReactDiff (LRS) at 0.29, yet Section 4.3.1 says ReactDiff's 0.26 is the highest. Table 2 shows Liu et al. (0.1675) and PerFRDiff (0.1368) above ReactDiff's 0.0940 on FRDiv, contradicting the diversity claim. Table 3 shows PerFRDiff at FRRea 47.62, better (lower) than ReactDiff's 66.72, while the text calls 66.72 the best. These are not edge cases; they are the central comparisons.\n\nThe 'Listener Reaction Shifts' variant that actually wins FRDist, FRDvs, and FRSyn is introduced in one sentence with no architecture, training schedule, or hyperparameters. That makes the best rows unreproducible and unattributable. The ablation is also under-discussed: removing the diffusion module raises FRCorr from 0.26 to 0.84. That is a real trade-off (diversity lost), but it undercuts the claim that the full model is the best balance.\n\nWhat the paper does well: the feature extraction is standard (ELIM, ME-GraphAU, ResMaskNet, wav2vec2), the MMT is described precisely, the DDIM equations are standard, and the evaluation uses the official REACT2024 protocol. The code link is included. The frozen-feature assumption is a fair concern but not the main problem.\n\nBottom line: the system is plausible and the reported numbers are probably real, but the paper as written does not support 'significantly outperforms'. A serious referee would ask for corrected claims, a defined LRS, and an honest treatment of the ablation. I'd send it to review with major revision, but I wouldn't cite the SOTA numbers until they are fixed.","headline":"The architecture is a plausible combination of known pieces, but the paper's own tables contradict its headline SOTA claims and the winning LRS variant is never defined.","tokens_in":19026,"tokens_out":3674,"would_cite":false,"duration_ms":54133,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:40:01.649495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}