{"id":"c5611610-8295-4123-8a97-d01da0ce984d","arxiv_id":"2507.22676","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A shared-compression MLP fusion with 32-head ensemble learning achieves MSE 0.1824, the top score in the AVI 2025 interview assessment track.","lead":"This system predicts five interview performance scores from a candidate's video, audio, and transcribed speech across six answers. It won first place in the AVI Challenge 2025 with an average mean squared error of 0.1824.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported first-place MSE is a single point estimate chosen from four pooling configurations by test-set performance, with no error bars, and Table 4's per-dimension validation values do not average to the stated validation MSE.","rationale":"The paper's central claim is a challenge leaderboard result, so the condition that must hold is that 0.18240 is a faithful, stable evaluation of the proposed architecture. That condition is least secure for three concrete reasons: (i) the final pooling configuration was chosen by test MSE among four options, making the reported point optimistic; (ii) no uncertainty is reported, and the margin to second place is smaller than the within-paper test spread across pooling choices; and (iii) the validation tables are numerically inconsistent, casting doubt on the paper's own metric computation. If a bootstrap or rerun revealed the difference to be within noise, the leaderboard rank might still be first, but the broader scientific claim of robustness and the value of 0.18240 as an unbiased estimate would fail. The proposed checks are feasible if the GitHub repository is complete: one inference rerun plus a bootstrap over the 130 test subjects. Because the paper already received a CONDITIONAL verdict and the concern does not prove the claim false, no change is needed; the condition is sharpened to require that the code reproduce 0.18240 and that the confidence interval exclude the runner-up score.","tokens_in":11101,"tokens_out":8474,"duration_ms":99514,"concrete_test":"Run the released inference code from https://github.com/MSA-LMC/365Aspects on the AVI2025-track-2 test features with the stated Max/Max pooling, 32-head ensemble, and mean-pooling over responses; recompute the average of the five per-dimension MSEs and compare with 0.18240. Then, using the official test predictions or rerunning with 10 random seeds, take 10,000 bootstrap resamples over the 130 subjects to obtain a 95% confidence interval for the mean MSE and for the difference versus the runner-up 0.18510. If the reproduced MSE differs by more than 0.00270, or if the confidence interval for the difference includes zero, the first-place claim is not established beyond selection and noise. Also verify whether the Table 4 per-dimension validation values average to the Table 2 validation MSE; currently they average to 0.15746, not 0.16810.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the claim 'first place with MSE 0.18240' over the runner-up 0.18510, the paper needs the test-set number to be a reliable evaluation of the proposed pipeline. Three things undercut this. First, Section 4.3 and Table 2 select the final Max/Max pooling configuration because it achieved the best test MSE (0.18240), not because it was best on validation (Mean/Max had 0.16730). The reported number is therefore the extremum of a small search, not an unbiased estimate of the method's test performance. Second, no variance or confidence interval is given for the 130-subject test set; the margin over second place (0.00270) is smaller than the spread among the four pooling configurations in Table 2 (range about 0.00607), so the ranking could change under resampling or seed variation. Third, the paper's own validation arithmetic does not close: Table 4 per-dimension validation MSEs for HFUT-VisionXL average to 0.15746, while Table 2 gives 0.16810 for the final configuration. If the evaluation tables are not internally consistent, the headline number needs independent confirmation before it can be treated as reliable evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal framework for asynchronous video interview performance assessment, combining three modalities (video, audio, text), six interview responses per candidate, and five target evaluation dimensions. Modality-specific pretrained encoders produce features that are fused by a proposed Shared Compression Multilayer Perceptron (MSCMLP), followed by an ensemble of 32 regression heads whose outputs are averaged across responses. The authors report a test MSE of 0.1824, claiming first place in the AVI Challenge 2025 track, and provide an ablation study over K-fold cross-validation, number of ensemble heads, and modality pooling strategies.","tokens_in":11379,"tokens_out":4810,"duration_ms":54906,"significance":"If the reported result is reliable, the paper is a practically useful challenge-report contribution: the MSCMLP fusion idea is simple and clearly motivated, the equations describing it are internally consistent, and the system reportedly outperforms several competing teams on a shared benchmark. The availability of code is a concrete reproducibility asset. However, the headline claim rests on a single test-set number that was selected over four pooling configurations by test performance, with no variance estimates or significance tests, and several internal numerical inconsistencies in the evaluation tables. The significance is therefore conditional on the authors tightening the experimental reporting.","major_comments":[{"comment":"The final configuration (SigLIP2 Max pooling, emotion2vec Max pooling) is selected because it achieved the best test MSE (0.1824), whereas the validation MSE is better for the Mean/Max configuration (0.1673). Thus the reported 0.1824 is the extremum of a four-way search on the test set, not an unbiased estimate of the pipeline's test performance. Moreover, the margin over the runner-up in Table 3 (0.00270) is smaller than the spread among the four pooling configurations in Table 2 (about 0.00607), and no repeated-seed variance, confidence interval, or significance test is provided. To support the first-place claim, please report the selection procedure, give multi-seed means and standard deviations, and state whether the test labels were accessed only once.","section":"§4.3, Tables 2 and 3"},{"comment":"The per-dimension validation MSEs for HFUT-VisionXL in Table 4 (0.1678, 0.1619, 0.1567, 0.1403, 0.1606) average to 0.15746, which does not match the validation MSE of 0.1681 reported for the final configuration in Table 2. Additionally, the claimed test improvement from K-fold cross-validation in Table 1 is 0.009598, but the numbers in the table imply 0.193083 − 0.184688 = 0.008395. These arithmetic inconsistencies must be resolved before the evaluation can be considered reliable.","section":"§4.3, Table 4 vs. Tables 1 and 2"},{"comment":"The experimental setup assigns dropout rates to a 'feature adapter,' a 'pure text processing module,' and a 'temporal control module,' but none of these components is defined in Section 3 or shown in Figure 2. The K-fold ablation in Section 4.3 never states the value of K, nor how cross-validation is integrated into training and final prediction. Without these details, the method and the claimed benefit of K-fold cross-validation are not reproducible.","section":"§4.2 and §4.3"},{"comment":"The final prediction is an unweighted mean over the six responses and 32 heads. The paper provides no analysis or ablation on whether all responses and all heads are equally informative for each of the five dimensions. If some interview questions carry more signal, or if response order matters, this equal-weight pooling discards information and the reported MSE depends on that choice. Please add an ablation or at least a discussion of per-response and per-head contributions to justify the pooling strategy.","section":"Eq. (11) and §4.4"}],"minor_comments":[{"comment":"The optimizer is written as 'Adaw,' which appears to be a typo for 'AdamW' or 'Adam'; please correct.","section":"§4.2"},{"comment":"Dimension names are inconsistent with Section 1: 'Collegiality' appears as 'Collectiveness,' 'Social versatility' as 'Social Diversity,' and 'Overall hireability' as 'Overall Employability.' Please align the terminology.","section":"Table 4"},{"comment":"The related-work section claims the method demonstrates 'superior generalization performance and prediction accuracy on multiple standard multi-input multi-label regression datasets,' but the paper evaluates only the AVI2025 dataset; please revise this claim.","section":"§2"},{"comment":"The phrases 'significantly outperforming' and 'significantly superior' are used without statistical tests; please temper them unless significance testing is added.","section":"§1 and Abstract"},{"comment":"The word 'learanble' appears in the description of the second linear layer; please fix the typo.","section":"§3.2"},{"comment":"The K-fold column uses '×' and '✓,' but the table caption does not define these symbols or state the K value.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short challenge report with a plausible and internally consistent technical core, but the headline result is not yet backed by statistically sound evidence: the final test MSE is selected over a small set of pooling configurations, the margin over second place is small, and the evaluation tables contain arithmetic inconsistencies. These issues are fixable with additional experiments and clarifications, so I recommend major revision rather than rejection. The editor may also wish to verify the challenge ranking with the AVI Challenge organizers, since the 'first place' claim is external to the manuscript's own evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid challenge-system paper, not a methodological breakthrough. The MSCMLP idea—modality-specific keys, shared value vectors—is a neat way to tie together SigLIP2, emotion2vec, and SFR-Mistral features, and the two-level ensemble (32 heads, then mean over 6 responses) is simple and defensible. The math in Section 3 is internally consistent, and the ablations show the expected trends. The reported first place on AVI 2025 with MSE 0.1824 is plausible, and the code is linked, which is more than most challenge write-ups do.\n\nThe soft spots are real and mostly about the headline number. The final Max/Max pooling configuration was chosen because it had the best test MSE (0.1824), not the best validation MSE (Mean/Max had 0.1673). That makes the reported test number the extremum of a four-way search, not an unbiased estimate. With 130 test subjects and no error bars, the 0.0027 margin over second place is within the spread of the four pooling configurations (range about 0.006). So the ranking should be treated as provisional.\n\nThere's also an internal inconsistency: Table 4's per-dimension validation MSEs for HFUT-VisionXL average to 0.1575, while Table 2 reports 0.1681 for the same final configuration. That discrepancy needs an explanation before the validation numbers can be trusted. Minor issues: the 'feature adapter' and 'temporal control module' referenced in Section 4.2 are never defined in the method, the K in 'K-fold' is never stated, and 'Adaw optimizer' is presumably a typo.\n\nIf you need this number for a comparison, treat it as conditional on running their code or re-evaluating on a held-out split. The architecture itself is worth a look for anyone doing multimodal fusion with heterogeneous embeddings; the shared-compression MLP is a clean alternative to more expensive cross-attention.\n\nRecommendation: send it to review. It's a legitimate challenge entry with a specific, checkable claim and public code. The referee should ask for error bars, a clear statement of how the pooling configuration was chosen, and a fix for the Table 2/Table 4 arithmetic.","headline":"A challenge-winning system with a clean architecture and a credible engineering win, but the headline MSE is the best of four test-set selections and the validation tables don't add up.","tokens_in":11878,"tokens_out":2100,"would_cite":false,"duration_ms":23495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a shared-compression MLP fusing video, audio, and text, plus a 32-head ensemble averaged over six answers, achieves an MSE of 0.1824 and first place in the AVI Challenge 2025.","keywords":["multimodal interview assessment","shared compression multilayer perceptron","multimodal fusion","multi-label regression","ensemble learning","AVI Challenge 2025"],"falsifier":"On the same AVI 2025 test split, replace the equal-weight average in Eq. (11) with weights learned per response, per dimension, or per head; if any such weighted variant lowers the multi-dimensional average MSE below 0.1824, the paper's pooling strategy is not the optimal use of its own representations, and its headline result would be tied to that choice.","tokens_in":10936,"feed_emoji":"🎤","tokens_out":10057,"duration_ms":97606,"temperature":0.7,"pith_summary":"This paper tries to show that interview performance can be scored automatically at a level that won the AVI Challenge 2025, by fusing three modalities (video, audio, and text) for each of six answers and predicting five job-relevant dimensions. The central move is to reinterpret a multilayer perceptron as an attention-like module and share its 'value' basis vectors across modalities, so all inputs are grounded in one low-dimensional space. On top of that, 32 regression heads per answer are averaged, and then the six answers are averaged, giving a multi-dimensional mean squared error of 0.1824 on the test set. A fair reader would come away with a concrete design that is simple, fully specified, and reproducible, and that outperforms the other six listed challenge systems.","feed_headline":"Shared-compression MLP tops AVI 2025 interview scoring with MSE 0.1824","feed_subtitle":"Video, audio, and text ratings for five job traits come from one shared basis layer plus a 32-head ensemble.","key_machinery":"The load-bearing mechanism is the Multimodal Shared Compression Multilayer Perceptron (MSCMLP), a variant of the classic MLP treated as an attention-like module. The first linear layer of each modality, followed by GeLU, produces activation scores; the second linear layer is a single set of learnable basis vectors shared by all three modalities, and each modality's output is the weighted sum of those basis vectors. Because the output dimension of the shared basis is set below the input dimensions, the fusion both aligns and compresses the 1152-dimensional video, 768-dimensional audio, and 4096-dimensional text embeddings into a unified space. The second mechanism is the two-level ensemble: each of six responses is scored by 32 parallel regression heads, the heads are averaged per response, and the response averages are mean-pooled to produce the final five scores.","core_discovery":"The paper's central claim is that its proposed framework—modality-specific extractors (SigLIP2 for video, emotion2vec-plus-seed for audio, SFR-Embedding-Mistral for text), a Shared Compression Multilayer Perceptron for fusion, and a two-level ensemble of 32 regression heads with response-level mean pooling—achieves state-of-the-art performance on the AVI 2025 interview assessment test set, with an average MSE of 0.1824 across integrity, collegiality, social versatility, development orientation, and overall hireability. The authors interpret the MLP's second linear layer as a set of shared basis vectors: each modality computes activation scores against its own first-layer weights, and those scores pick out a weighted combination of the same basis, producing a compressed common embedding. They report that the combination of K-fold cross-validation, max-pooling for both video and audio features, 32 ensemble heads, and mean-pooling across the six responses yields their best test result, and that their validation-set performance is superior to the AVI baseline on four of the five dimensions.","pith_inferences":["Because Eq. 11 weights every response and head equally, a likely untested improvement is to learn per-question or per-dimension weights; if the six questions differ in how much they reveal about, say, integrity, that pooling is discarding signal the same architecture could use.","The MSCMLP attention analogy suggests a direct comparison with low-rank bilinear pooling or cross-attention fusion on the same features; the paper's result implies the benefit may come from compression and a shared basis rather than explicit pairwise interaction, but the paper does not run that comparison.","The test-set leaderboard gap to the second-place system is 0.0027 MSE; a resampling or repeated-split analysis would show whether that margin is stable, something the paper does not report.","A practical extension would be to apply the same shared-basis fusion to asynchronous video interview screening at scale, where the six responses are fixed and the five dimensions are known to raters — the framework's simplicity keeps deployment costs low."],"forward_implications":["If the 0.1824 test MSE holds, a competitive multimodal interview scorer can be built from frozen pretrained encoders plus a single small fusion MLP, with no fine-tuning of the encoders and no cross-modal attention.","The shared-basis design implies one learned linear layer is enough to project heterogeneous embeddings into a common lower-dimensional space where a prediction head can read them, which transfers to other multi-modal regression settings.","The two-level averaging implies that repetition — multiple answers per candidate and multiple heads per answer — is itself treated as an accuracy booster, exploiting variance reduction rather than more complex modeling.","Per-dimension results on the validation set indicate the largest gains over the AVI baseline are in collegiality, social versatility, development orientation, and overall hireability, while the baseline retains the edge on integrity."],"supporting_citations":[{"why":"Supplies the visual feature extractor (SigLIP2) that encodes video frames for the video modality.","marker":"[40]"},{"why":"Supplies the self-supervised speech emotion model (emotion2vec-plus-seed) that extracts audio features from spectrograms.","marker":"[28]"},{"why":"Supplies the LLM-based text embedder (SFR-Embedding-Mistral) that encodes transcribed responses for the text modality.","marker":"[30]"},{"why":"Provides the projective-network inspiration for grouping features by modality and using shared feature projection in MSCMLP.","marker":"[32]"},{"why":"Defines the classic MLP and feedforward network structure that MSCMLP and the 32 regression heads build on.","marker":"[35]"},{"why":"Provides the automatic speech recognition used to transcribe audio into text, creating the text input stream.","marker":"[47]"},{"why":"Defines the MSE criterion used by the AVI Challenge to score predictions, the metric on which the first-place claim rests.","marker":"[18]"}],"fun_headline_variants":["Multimodal fusion wins AVI 2025: shared MLP reads unspoken cues","365 aspects, one shared layer: AVI 2025 champion at MSE 0.1824","MLP shared basis wins AVI 2025: fuses video, audio, text for 5 traits","Unspoken cues exposed: shared MLP wins AVI 2025 interview challenge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The final prediction is a simple average of every answer and every scoring head, so the whole result depends on those answers and heads being equally informative for each of the five dimensions.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal fusion wins AVI 2025: shared MLP reads unspoken cues","365 aspects, one shared layer: AVI 2025 champion at MSE 0.1824","MLP shared basis wins AVI 2025: fuses video, audio, text for 5 traits","Unspoken cues exposed: shared MLP wins AVI 2025 interview challenge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3292,"prompt_tokens":1009,"completion_tokens":2283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2184}},"tokens_in":625,"tokens_out":2283,"duration_ms":14442,"temperature":1.0,"reasoning_tokens":2184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:23:37.823576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same AVI 2025 test split, replace the equal-weight average in Eq. (11) with weights learned per response, per dimension, or per head; if any such weighted variant lowers the multi-dimensional average MSE below 0.1824, the paper's pooling strategy is not the optimal use of its own representations, and its headline result would be tied to that choice.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-based text embedder (SFR-Embedding-Mistral) that encodes transcribed responses for the text modality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the projective-network inspiration for grouping features by modality and using shared feature projection in MSCMLP."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the classic MLP and feedforward network structure that MSCMLP and the 32 regression heads build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the automatic speech recognition used to transcribe audio into text, creating the text input stream."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MSE criterion used by the AVI Challenge to score predictions, the metric on which the first-place claim rests."}],"review_version":1}