{"id":"2cca48d6-4b62-47f5-8162-2038da47d01e","arxiv_id":"2508.06800","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HARDY-MER is a retrieval-augmented curriculum training method that gives harder samples more retrieved support samples, improving missing-modality emotion recognition on average across IEMOCAP and CMU-MOSEI.","lead":"This paper proposes HARDY-MER, a training framework that measures how hard each sample is to process when a modality is missing, then uses that difficulty to retrieve a variable number of similar support samples during training. The goal is to improve emotion recognition on test inputs with missing audio, text, or video, and the authors report gains over five baselines on IEMOCAP and CMU-MOSEI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-time retrieval can include the query's own full-modality record, leaking the missing modality into the curriculum; since retrieval is dropped at inference, the reported SOTA gains may be a train/test shortcut.","rationale":"The paper's central claim is that HARDY-MER improves robustness under missing modalities. For that claim to hold, the training procedure must not give the model access to the missing modality through the retrieval mechanism. The described pipeline does not state that the query is removed from the FAISS index, and the retrieved support samples are full multimodal records. This is the least secure condition in the argument. The reader's weakest assumption already pointed at a retrieval shortcut that disappears at inference; I sharpen it to a concrete self-retrieval mechanism that leaks the missing modality itself, not merely similar labels. The alternative concern about indirect hardness (Section 2.1 vs. Eq. 9) is real and worth fixing, but it concerns the conceptual validity of the hardness metric rather than directly explaining away the empirical gains. The self-retrieval issue, by contrast, threatens the validity of the reported comparisons. The proposed ablation would settle whether the gains survive when the shortcut is removed. Since the paper currently does not provide this evidence, conditional acceptance is appropriate: if the ablation shows the gains vanish, the central claim should be rejected; if they persist, the concern is resolved.","tokens_in":16671,"tokens_out":8079,"duration_ms":107007,"concrete_test":"Modify Step 2.2 to exclude the query sample (and any identical/near-identical records, e.g., the same conversational turn) from the FAISS index before retrieval, and also mask the missing modality in all retrieved support features so support samples never contain the modality absent from the query. Retrain HARDY-MER on IEMOCAP 4-class and compare the six per-condition WA/UA and the Average column of Table 1 against the reported values. If the improvements over MoMKE shrink to within noise or disappear, the reported SOTA rests on the self-retrieval shortcut rather than on learned missing-modality robustness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.3.1 builds modality feature databases from the training data. In Section 3.3.2, each training sample queries these databases with its available-modality embeddings; the text never excludes the query itself from the index. Because the query's own features have zero L2/inner-product distance, self is always ranked first, and with k=5 and k'=ceil(h·k) at least one support sample is always used. Retrieved candidates are drawn from all three modality databases, so the self-support contains the very modality that is marked missing in the input. Section 3.3.3 then concatenates these support features during training and states that inference uses no retrieval. Consequently the model can satisfy the reconstruction/classification objectives by copying the missing modality from the retrieved self/near-duplicate, rather than by learning to predict from available modalities. The w/o retrieval features ablation removes all retrieval and therefore cannot separate this leakage from any legitimate benefit of augmentation. This is a concrete train/test distribution mismatch that directly undermines the central missing-modality robustness claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HARDY-MER, a training-time framework for multimodal emotion recognition with missing modalities. It first estimates per-sample hardness using reconstruction errors (\"direct hardness\") and cross-modal mutual information (\"indirect hardness\"), then uses retrieval from modality-specific FAISS indexes to augment each training sample with a hardness-dependent number of support samples, with more supports for harder samples. The final model is trained on the original incomplete input plus the retrieved supports; at inference no retrieval is used. Experiments on IEMOCAP (4-class and 6-class) and CMU-MOSEI report improved weighted accuracy/accuracy over baselines across six missing-modality settings, with a claimed new state of the art.","tokens_in":16958,"tokens_out":5693,"duration_ms":70376,"significance":"If the results are valid, the combination of hardness-aware curriculum learning with retrieval augmentation is a novel and practically interesting idea for missing-modality MER, and the promised code release would facilitate reproducibility. The paper includes ablations for each hardness component, the adaptive curriculum, retrieval features, and index construction choices. However, two load-bearing issues currently undermine the central claim: (1) the retrieval database is not protected against self-retrieval, so the training signal can leak the missing modality directly into the input, and (2) the signature of the indirect-hardness term contradicts the paper's own definition of hardness. Both issues require re-running experiments after fixing the protocol.","major_comments":[{"comment":"The feature database is built from training samples (Step 2.1), and Step 2.2 queries it with the same sample's available-modality embeddings without excluding the query itself. Since Eq. (18) gives k' = ceil(h·k) with h in (0,1) and k=5, at least one retrieved sample is always used, and the query's own all-modality record is the nearest neighbor. The support set therefore supplies exactly the modality that is marked missing at training time, while §3.3.3 states inference uses no retrieval. The model can minimize reconstruction/classification loss by copying the missing modality from the retrieved self rather than by learning to predict from available modalities. This is a train/test distribution mismatch that directly affects the central missing-modality claim. The w/o retrieval features ablation removes all support and cannot isolate this leakage. The authors should exclude self from th","section":"§3.3.2–3.3.3"},{"comment":"The motivation states that weak cross-modal consistency makes samples hard, but Eq. (9) defines indirect hardness as the sum of positive mutual information values, so higher MI yields higher h_ind, i.e., stronger consistency is scored as harder. Eq. (13) then minimizes -h_ind, which maximizes MI—consistent with treating high MI as desirable, not hard. This sign contradiction means the hardness score is not a faithful implementation of the paper's own hardness concept, and the dynamic curriculum may emphasize exactly the samples the authors intend to down-weight. The definition of h_ind or the motivation needs to be changed, and the hardness-adaptive component must be re-evaluated.","section":"§2.1, Eq. (9), Eq. (13)"},{"comment":"The table claims significance with '*' and 'T-test' but no standard deviations, confidence intervals, or number of runs are reported anywhere. A paired t-test on the Average column requires multiple independent runs; with a single run per condition, the p-value cannot be computed. Please report mean±std over at least 5 random seeds and full test details. Without this, the 'consistently outperforms' claim is not statistically supported.","section":"Table 1"}],"minor_comments":[{"comment":"The code URL is inconsistent: the abstract gives https://github.com/HARDY-MER/HARDY-MER while the full text gives https://github.com/AI-S2-Lab/HARDY-MER. Please unify.","section":"Abstract / Footnote 1"},{"comment":"The text says 'retrieved the top 1502 most similar samples' — this is likely a typo for 'top 150' or 'top 1500'; please correct.","section":"§4.5"},{"comment":"The mutual information is computed via entropies H(f_p), H(f_q), H(f_{p,q}), but no estimator is specified for high-dimensional entropy. Since Eq. (8) is a central component of indirect hardness, a concrete estimation method (e.g., binning, kernel density, or a neural estimator) should be stated.","section":"Eq. (8)"},{"comment":"The claim of being 'the first to integrate retrieval and curriculum learning' and 'the first work to apply RAG technology to multimodal emotion recognition' is stronger than what the cited literature supports; please soften or add a more precise comparison to prior retrieval-augmented training methods.","section":"§5"},{"comment":"The hyperparameter ablation explores only one parameter at a time and reports no significance. Since alpha1, alpha2, beta, and k are central to the dynamic curriculum, a sensitivity analysis with more values and multiple seeds would be more convincing.","section":"Table 3"},{"comment":"The 'w/o retrieval features' row removes all retrieval, effectively testing the whole retrieval curriculum; it cannot separate the effect of self-retrieval leakage from the benefit of external similar samples. This is related to Major Comment 1 and should be discussed.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The self-retrieval problem is, in my view, the most serious issue: if the training procedure feeds the query's own complete-record back through the concatenated support set, the reported improvements may simply reflect an oracle shortcut that vanishes at test time. The authors can address this by excluding the query (and possibly its near-duplicates) from the FAISS index and re-running all tables. The indirect-hardness sign error is also substantial and should be fixed before the hardness-aware curriculum can be meaningfully evaluated. I would not recommend acceptance without these changes, but the paper's core idea is worth pursuing if the revised experiments still show gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe paper is a sensible integration of ideas that mostly already exist: hard-sample mining, retrieval augmentation, curriculum learning. What is genuinely new is the packaging—a multi-view hardness score (reconstruction error plus cross-modal mutual information) and a dynamic curriculum where the number of retrieved support samples scales with estimated hardness. The experiments are broad (two datasets, six missing-modality settings, multiple ablations), and the authors are unusually careful about the retrieval index choices and about freezing the hardness module after training. For that reason I wouldn't call this a trivial paper.\n\nThe problem is the central robustness claim is probably contaminated by a train-time leakage. The retrieval database is built from the training set, and the query sample itself is not excluded. Since k'=ceil(h*k) and h>0, at least one support sample is always used. With k=5, self is ranked first. So during training the model receives its own full-modality record, including the modality that is supposed to be missing. At inference, retrieval is switched off. That is a textbook train/test shortcut. The w/o retrieval features ablation removes all retrieval and therefore cannot show whether the benefit is from learning robust unimodal features or from copying the missing modality from a near-duplicate. The stress-test note lands; in fact it is even stronger because the 'support' sample is not just semantically similar, it is the exact same utterance.\n\nThere are smaller problems too. The indirect hardness formula contradicts the paper's own motivation: Section 2.1 says weak cross-modal consistency should make a sample hard, but Eq. 9 adds mutual information, so high MI is called hard. That is a sign error in the conceptual framing, even if the ablation still shows the module helps. The main table has no error bars or run counts, and the hyperparameters in Eq. 10 plus k are chosen on the benchmark, so the gains of 0.01–0.05 average WA are not yet known to be robust.\n\nWho this is for: people working on missing-modality MER and retrieval-augmented training. It deserves a serious referee, but not acceptance as a SOTA claim in its current form. The fix is straightforward—exclude the query's own index entry, or test a no-self-retrieval variant and see how much of the gain survives. As is, I would not rely on the headline numbers.","headline":"A sensible integration of hardness-aware curriculum and retrieval, but the main robustness claim is undermined by a train-time self-retrieval leak.","tokens_in":17370,"tokens_out":2396,"would_cite":false,"duration_ms":29476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes HARDY-MER, a training framework that improves multimodal emotion recognition when audio, text, or video inputs are missing by estimating per-sample difficulty and retrieving similar support examples for hard cases, report","keywords":["multimodal emotion recognition","missing modalities","curriculum learning","hard sample mining","retrieval-augmented training","mutual information","reconstruction error","IEMOCAP"],"falsifier":"Train HARDY-MER on a variant where the retrieved support samples are replaced by random samples of the same emotion class (keeping k' the same); if average accuracy does not drop, the hardness-aware retrieval content is not responsible for the reported gains. Alternatively, swap the training retrieval database for one built from a different dataset; if gains vanish, the method does not generalize to unseen distributions.","tokens_in":16581,"feed_emoji":"🎭","tokens_out":4399,"duration_ms":42183,"temperature":0.7,"pith_summary":"The paper proposes a training method called HARDY-MER for multimodal emotion recognition when some input modalities (audio, text, video) are missing at inference. The central idea is to score how hard each training sample is to learn—by how well available modalities reconstruct the missing one and how much information modalities share—then to give harder samples more help during training by retrieving and appending similar labeled examples. The authors report that this consistently outperforms existing missing-modality methods on IEMOCAP and CMU-MOSEI, with average weighted-accuracy gains of roughly 4.4 points, 3.0 points, and 1.4 points over the best competing method in three settings. The method does not need retrieval at test time; the trained model predicts from whatever modalities are present.","feed_headline":"Hardness-aware training beats missing-modality emotion baselines","feed_subtitle":"Retrieving similar examples for hard samples lifts accuracy on IEMOCAP and CMU-MOSEI.","key_machinery":"The multi-view hardness score h = (1 + exp(-$\\beta$ * (alpha1 * h_dir + alpha2 * h_ind)))^-1, combining direct reconstruction error and indirect mutual information; and the hardness-aware retrieval scheduler k' = ceil(h*k), which determines how many similar samples are appended to each training input.","core_discovery":"HARDY-MER claims that sample-specific training hardness, estimated from reconstruction error and cross-modal mutual information, can be turned into a dynamic curriculum that improves emotion recognition under missing modalities. For each input, the framework computes a unified hardness score in (0,1), retrieves the most semantically similar samples from per-modality feature banks, and keeps only k' = ceil(h*k) of them, so harder samples get more support. These support samples are concatenated with the input during training, and the model is trained with classification and reconstruction losses. The authors report new state-of-the-art average results under six missing-modality settings, with","pith_inferences":["The paper does not test whether the model learns to use retrieved samples as a crutch rather than as support; an editor's inference is that the strongest validation would compare against retrieving random or same-label samples with the same k' schedule.","A natural extension is to apply the same hardness-aware retrieval curriculum to other incomplete-input tasks such as multimodal sentiment analysis with noisy or corrupted modalities, or to audiovisual speech recognition.","The mutual information approximation via cross-attention is a practical choice; a stricter estimator might change hardness rankings, so results could be sensitive to that approximation."],"forward_implications":["If the claim holds, training-time retrieval augmentation is enough to improve missing-modality accuracy without changing the inference model or requiring external data at test time.","Per-condition metrics improve across all six missing-modality settings on IEMOCAP and on most settings on CMU-MOSEI; average gains are statistically significant (p<0.05) by the paper's T-test.","The largest gains occur under video-only input, suggesting the method helps most when the remaining modality is least informative.","Removing either hardness component or fixing k' degrades performance, implying the adaptive curriculum is the active ingredient."],"supporting_citations":[{"why":"Supplies the cross-attention fusion approach used to approximate mutual information between modalities for indirect hardness.","marker":"[10]"},{"why":"Provides the reconstruction-error-based hard-sample mining idea that motivates direct hardness.","marker":"[39]"},{"why":"The strongest competing baseline (MoMKE) and a source of publicly available features used in the experiments.","marker":"[50]"},{"why":"Baseline method (CIF-MMIN) using modality-invariant features, which the proposed method compares against and extends.","marker":"[23]"},{"why":"Baseline method (GCNet) for graph-completion-based missing-modality learning and a source of features for the benchmarks.","marker":"[18]"},{"why":"Baseline method (MMIN) for missing modality imagination, against which HARDY-MER is evaluated.","marker":"[55]"}],"fun_headline_variants":["Hardness-aware curriculum beats missing-modality emotion baselines","Dynamic difficulty scheduling improves missing-modality emotion recognition","Retrieval-based sample weighting lifts multimodal emotion accuracy","Hardness-driven training excels under missing emotion modalities"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"During training the model sees retrieved similar samples appended to its input, but at inference retrieval is switched off, so the claimed gains depend on the model learning transferable unimodal and multimodal features rather than exploiting the retrieved context as a shortcut.","fun_headline_variants_meta":{"raw":{"variants":["Hardness-aware curriculum beats missing-modality emotion baselines","Dynamic difficulty scheduling improves missing-modality emotion recognition","Retrieval-based sample weighting lifts multimodal emotion accuracy","Hardness-driven training excels under missing emotion modalities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1008,"prompt_tokens":731,"completion_tokens":277,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":475,"tokens_out":277,"duration_ms":3657,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:31:04.256678+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HARDY-MER on a variant where the retrieved support samples are replaced by random samples of the same emotion class (keeping k' the same); if average accuracy does not drop, the hardness-aware retrieval content is not responsible for the reported gains. Alternatively, swap the training retrieval database for one built from a different dataset; if gains vanish, the method does not generalize to unseen distributions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cross-attention fusion approach used to approximate mutual information between modalities for indirect hardness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reconstruction-error-based hard-sample mining idea that motivates direct hardness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The strongest competing baseline (MoMKE) and a source of publicly available features used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline method (CIF-MMIN) using modality-invariant features, which the proposed method compares against and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline method (GCNet) for graph-completion-based missing-modality learning and a source of features for the benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline method (MMIN) for missing modality imagination, against which HARDY-MER is evaluated."}],"review_version":1}