{"id":"e71134a6-8c3d-4495-b706-ac0de81fbb0e","arxiv_id":"2506.09745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces MMHCL, a setting where modalities train on heterogeneous category sets, and CSCF, a model using semantic alignment, uncertainty-based dominance selection, and class-similarity fusion to recognize the complete class space.","lead":"This paper defines a new multimodal classification setting, MMHCL, in which different modalities are trained on different category sets but must jointly recognize the full union of categories at test time. It proposes CSCF, a method that aligns features to a semantic space, uses uncertainty to pick the dominant modality, and fuses auxiliary predictions via class similarity, reporting large gains over baselines on three audio-video benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise is that u* in Eq. (7) reliably flags which modality saw the class, yet it is validated on only 30 samples; a material misselection rate would invalidate the claimed accmix gains.","rationale":"The paper proposes a well-motivated new setting, MMHCL, and the method is clearly described, with the semantic-alignment idea being a reasonable way to enable zero-shot transfer to unseen classes. The reader's conditional verdict is appropriate. I agree with the reader's weakest_assumption: the uncertainty heuristic is the load-bearing element of the claimed superiority. The paper's own Appendix E and the 30-sample validation make this explicit. I also note the absence of code and implementation details for the ensemble OSRS modules, and the lack of significance tests, but those are secondary to the selection rule. A full-test selection analysis and an oracle-fusion comparison would settle whether the central claim holds. Since the reader already conditioned the verdict on verification, I do not change the verdict; however, if the proposed test shows a large oracle/u* gap, the verdict should be lowered to reflect that the measured gains rest on an unreliable mechanism.","tokens_in":16039,"tokens_out":9542,"duration_ms":122767,"concrete_test":"On ActivityNet (or all three datasets) with the released code, compute the dominant-modality selection accuracy of Eq. (7) over every test sample in As+Bu, Au+Bs, and Aall+Ball, using the known training-category split as oracle; report a confusion matrix of u*-selected vs oracle-selected modalities. Then fix top-k to one shared value and compare final accmix under three decision rules: Eq. (7) selection, oracle selection, and simple average fusion. If the oracle/u* agreement is below ~90% or the oracle-vs-u* accmix gap exceeds ~5 points, the uncertainty rule is the bottleneck and the Table 1 gains should be re-evaluated; if the gap is within noise, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accmix advantage in Table 1 rests on the fixed rule in Eq. (9): the modality with lower uncertainty u* = Inc* + dif* is chosen as dominant and the other modality only refines that decision. Figures 7 and 9 provide the only evidence for this rule, and each uses only 30 randomly selected samples; no full-test selection accuracy or confusion matrix is reported. This is not an incidental detail: Appendix E documents that a modality which never saw a class can be overconfidently wrong for a portion of samples, and the paper's dif* is normalized entropy rather than an explicit inter-modality difference or a correctness-calibrated score. If the entropy/ensemble-disagreement proxy misidentifies the dominant modality on even a modest share of As+Bu/Au+Bs samples, the fusion in Eq. (9) can add wrong-class mass to the dominant logits, and the reported +10.23/+10.77/+3.30 accmix gains could shrink or vanish. The absence of error bars and the per-dataset selection of top-k make it impossible to tell whether those gains are robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multi-Modal Heterogeneous Category-set Learning (MMHCL), where each modality is trained on a different subset of categories and the model must classify the union of categories at test time. It proposes CSCF, composed of (i) an Open Set Recognition via Semantics module that maps modality features into a shared CLIP/LLM semantic space and classifies by scaled cosine similarity, (ii) a Dominant Modality Selection Strategy that combines intra-modality ensemble entropy and inter-modality entropy to pick a dominant modality, and (iii) a Class Similarity-guided Multimodal Fusion module that prunes a class-similarity matrix to top-k and adds the auxiliary modality's similarity-weighted logits to the dominant modality's logits. Experiments on ActivityNet, UCF, and VGGSound compare against seven ZSL/MML/IMV baselines and report accmix gains of 10.23, 10.77, and 3.30 percentage points, with ablations showing each module contributes incrementally.","tokens_in":16275,"tokens_out":5873,"duration_ms":63739,"significance":"The MMHCL setting is practically motivated and, among the cited baselines, not directly addressed: standard multimodal methods assume complete category coverage, and incomplete-multi-view methods handle only sample-level missing modalities. The method avoids circularity with the training data by using external CLIP/CLAP features and LLM-generated class descriptions, and the ablation study shows consistent incremental gains from OSRS, DMSS, and CSMF across three datasets. The claimed gains, if robust, would make this a useful baseline for research on combining modality data with heterogeneous label sets. However, the validation of the dominant-modality selection mechanism is thin, and the absence of statistical error bars and full hyperparameter disclosure currently prevents the central claim from being fully supported.","major_comments":[{"comment":"The DMSS decision rule is load-bearing: Eq. (9) hard-switches the fusion on the comparison of uA and uB, where u* = Inc* + dif*. The only validation of this rule is the qualitative inspection of 30 randomly selected samples in Figures 7 and 9; no full-test selection accuracy, per-case confusion matrix, or comparison against an oracle selector is reported. The paper's own Appendix E additionally shows that a modality that never saw a class can produce overconfident predictions on some samples, which is exactly the failure mode that would corrupt the hard switch. Please report the dominant-modality selection accuracy on the full test sets, and provide an ablation that replaces Eq. (9) with an oracle selector to quantify the headroom lost to misselection.","section":"§4.4, Eq. (7)–(9), Appendix E"},{"comment":"The central claim that CSCF 'significantly outperforms' state-of-the-art methods rests on single numbers with no error bars, no number of seeds, and no significance tests. Since the reported accmix improvements differ across datasets (10.23, 10.77, and 3.30 points) and the top-k hyperparameter is tuned per dataset, the reader cannot distinguish a robust advantage from favorable hyperparameter selection. Please report mean and standard deviation over at least three random seeds per dataset and, if appropriate, paired significance tests or confidence intervals.","section":"Table 1"},{"comment":"Several components that the fusion equations depend on are underspecified. The number K of OSRS modules, the logit scale γ in Eq. (2), and the top-k values in the pruned similarity matrices S* are never given; Appendix D only lists feature dimensions, optimizer, epochs, and batch size. Moreover, Eq. (8)–(9) do not state whether the rows of S* are normalized (e.g., softmax over the retained neighbors) or whether the diagonal is kept, which changes the scale of the added term relative to the dominant logits. Please specify all hyperparameter values per dataset, describe the K OSRS module architectures (Appendix B mentions 'four OSRS modules with different architectures' without details), and give a sensitivity analysis for top-k and γ on all three datasets.","section":"§3.3, Appendix D"}],"minor_comments":[{"comment":"The heading 'Consturction of DMSS' should read 'Construction of DMSS'.","section":"§3.2"},{"comment":"The heading 'Implement Details' should read 'Implementation Details', and 'Muti-View' should be 'Multi-View'.","section":"§4.1"},{"comment":"The notation 'Aall+Ball' appears in Table 1 but is not defined in Appendix C, which defines only As, Bs, Au, Bu, As+Bu, and Bs+Au; please define it explicitly.","section":"Table 1 and Appendix C"},{"comment":"Figure 11 is based on twenty randomly sampled instances and reports no axis label for the horizontal axis beyond 'Sample Index'; please clarify the sampling procedure and consider reporting a larger sample or error bars.","section":"Appendix E, Figure 11"},{"comment":"The input line of Algorithm 1 lists only {X A, YA s } before the colon, whereas the problem formulation and surrounding text include modality-B data; please make the notation consistent.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a practically relevant setting and large claimed gains, but the uncertainty-selection evidence is the main technical risk. I would like the revision to include full-test selection accuracy for the DMSS rule, multi-seed statistics, and complete disclosure of the top-k, K, γ, and OSRS architecture choices. Please also ask the authors to check the 'first study on MMHCL' claim against label-missing multi-label and incomplete multi-view literature, since the boundary with those settings is not fully discussed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper introduces a genuinely new setting—MMHCL—where different modalities are trained on disjoint or partially overlapping category subsets and the model must classify the union at test time. That framing is practical and clearly distinguishes it from GZSL, standard multimodal learning, and incomplete-view classification. The method, CSCF, is a coherent pipeline of three known components—semantic alignment via CLIP/LLM features, ensemble-based uncertainty for dominant-modality selection, and class-similarity-guided fusion. Each component is motivated and the ablations show each adds value.\n\nThe empirical evaluation is the paper's real strength. Results on three datasets are large and consistent: gains of 10.23, 10.77, and 3.30 accmix points over the strongest baseline. The authors do not oversell: they report unseen-class performance for each modality, and the appendix includes an honest analysis of overconfidence in unseen modalities.\n\nThe soft spots are real but not fatal. The most important is statistical: no error bars, no significance tests, and the top-k hyperparameter is tuned per dataset. The stress-test concern about the uncertainty-based dominance rule (Eq. 7/9) is legitimate: the rule is validated on only 30 randomly selected samples per case, and the paper never reports full-test selection accuracy. If the entropy proxy mis-selects the dominant modality on even a modest share of test samples, the reported accmix gains could be smaller than claimed. That said, the ablation in Table 2 shows DMSS improves over simple averaging and CSMF improves further, so the pipeline is not a house of cards. I would want a full-test analysis of dominance-selection accuracy before trusting the exact numbers.\n\nMinor issues: implementation details for the ensemble modules are thin, and the supplementary code is promised rather than shipped. The citation pattern is fair; related work is covered and differentiated adequately.\n\nVerdict: this deserves a serious referee. The setting is meaningful, the method is sensible, and the results are promising enough to warrant a conditional accept with requests for error bars, full selection analysis, and code release. I'd bring it to a reading group.","headline":"A new and practical problem setting with a sensible first solution; the empirical gains are large, but the lack of error bars and the thin validation of the uncertainty-based dominance rule need referee attention.","tokens_in":16777,"tokens_out":1561,"would_cite":true,"duration_ms":18829,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows a multimodal classifier trained on partially overlapping audio and video label sets can recognize the union of classes via semantic alignment, uncertainty-based modality choice, and class-similarity fusion.","keywords":["multimodal classification","heterogeneous category sets","zero-shot learning","uncertainty estimation","class-similarity fusion","audio-visual recognition","semantic alignment","open-set recognition"],"falsifier":"Take a held-out set with at least a few hundred samples per class-side, compute fused top-1 accuracy, and re-compute it after forcing the higher-uncertainty modality to be dominant; if the forced-wrong choice does not lower accuracy substantially on classes where the entropy rule guessed wrong, the uncertainty claim is empty. More sharply: on samples whose true class was seen only by the modality with higher $u$, the fused accuracy should beat that modality alone; if it does not, the rule is falsified.","tokens_in":15867,"feed_emoji":"🎬","tokens_out":8515,"duration_ms":79610,"temperature":0.7,"pith_summary":"The paper proposes a new task format, Multi-Modal Heterogeneous Category-set Learning (MMHCL), in which each training modality sees only a subset of the class space, yet the resulting model must recognize the union of all classes at test time. It claims this is the first work to treat category-set mismatch across modalities as a learnable problem rather than an obstacle. The paper presents CSCF, which aligns each modality's features to a shared semantic space built from LLM-enriched class descriptions, uses ensemble uncertainty to pick the more reliable modality for each sample, and fuses the weaker modality through class-similarity reweighting. On ActivityNet, UCF, and VGGSound, the method reports mixed-scenario accuracy gains of 10.23, 10.77, and 3.30 percentage points over the strongest competing approach. If this holds, multimodal systems can be assembled from separately collected, separately labeled datasets in one training session.","feed_headline":"One training run fuses audio and video with mismatched label sets","feed_subtitle":"Class-similarity fusion with uncertainty selection beats top baselines by 10+ points on three audio-video benchmarks.","key_machinery":"The load-bearing machinery is the semantic space plus the uncertainty-decided fusion. Class names are expanded by a large language model into descriptive sentences and embedded with CLIP's text encoder to produce prototypical semantic features; each modality's backbone features are mapped into this space by a learned mapper, and classification scores are cosine similarities between mapped features and all class prototypes, forming the OSRS module. Four mappers per modality form an ensemble whose per-sample cross-entropy spread defines intra-modality inconsistency; combined with the entropy of averaged predictions across modalities, this yields an uncertainty $u^*$ used to declare the dominant modality. The CSMF module then prunes the class-similarity matrix to the top-$k$ neighbors per class and reweights the auxiliary modality's logits before adding them to the dominant logits, so the final decision is driven by the modality that was trained on the true class.","core_discovery":"The central claim is that category heterogeneity between modalities is not a failure case to be patched but a learnable setting. A model trained on modality A's subset of classes and modality B's subset can recognize every class across both modalities, provided three ingredients are present: a shared semantic space that carries class meaning, built from LLM-expanded class descriptions embedded by CLIP; a per-sample estimate of which modality is expert for the test class, computed as entropy-based uncertainty over an ensemble of sub-modules; and a fusion rule that lets the non-expert modality contribute only through class-similarity-weighted logits. The paper reports that this combination outperforms seven state-of-the-art methods from generalized zero-shot learning, multimodal classification, and incomplete multi-view learning, with the largest gains on scenarios where one modality must classify classes it never saw.","pith_inferences":["Because the uncertainty rule is a fixed statistic rather than a learned module, it should transfer to more than two modalities by taking the argmin over per-modality uncertainties; this follows naturally from the paper's design but is not tested there.","The class-similarity top-$k$ fusion suggests a testable extension: learn the per-class $k$ or weight the similarity matrix by dataset-specific confusions, which could recover performance lost when semantically close but visually distinct classes are neighbors.","A broader consequence the paper leaves implicit is that if LLM-enriched semantics are a sufficient bridge, datasets collected for different tasks can be combined without re-annotating labels, turning MMHCL into a dataset-engineering strategy.","The evaluation splits each dataset's classes evenly and strictly disjointly between modalities; the setting would be stress-tested by partially overlapping label sets, unequal class counts, and three or more modalities, which the paper does not experiment with."],"forward_implications":["A single CSCF training session can ingest audio-only and video-only datasets with different class lists and produce one classifier covering the union of both lists.","Semantic alignment through LLM-generated class descriptions gives each modality a path to classes it never saw, so unseen-class accuracy stops being zero for the non-expert modality.","Uncertainty-based dominant-modality selection suppresses unseen-class noise from the auxiliary modality, which the paper shows is the main failure of confidence-based selection in unimodal zero-shot baselines.","Class-similarity reweighting lets the auxiliary modality add informative logit mass to semantically related classes, improving fusion accuracy beyond simply averaging or picking the more confident model.","Existing multimodal, incomplete-view, and generalized zero-shot methods lose performance under MMHCL because they either require full category coverage or cannot fuse cross-modal information; the proposed protocol exposes that gap."],"supporting_citations":[{"why":"Supplies the large language model that expands raw class names into descriptive sentences, which anchor the shared semantic space.","marker":"[68]"},{"why":"Provides the CLIP/CLAP backbone features used for all three datasets and the CLIP text encoder used for semantic embeddings.","marker":"[13]"},{"why":"Motivates the open-set recognition view that maps features into a semantic space to reach unseen classes.","marker":"[69]"},{"why":"Motivates the ensemble of OSRS sub-modules whose disagreement feeds the uncertainty estimate.","marker":"[70]"},{"why":"Baseline generalized zero-shot method that the paper's method must beat in the MMHCL comparison.","marker":"[1]"},{"why":"Baseline trusted multi-view classifier representing the multimodal learning assumption of complete category coverage.","marker":"[2]"},{"why":"Baseline incomplete multi-view method that the paper improves on by the stated margins on ActivityNet.","marker":"[3]"},{"why":"Baseline zero-shot logit adjustment method that the proposed fusion is compared against.","marker":"[66]"}],"fun_headline_variants":["Fusing audio and video when each knows different classes","Class-similarity fusion learns from mismatched label sets","One training run recognizes all classes from hetero-labeled modalities","Uncertainty-driven fusion beats baselines in heterogeneous label sets","Shared semantic space bridges modalities with distinct categories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that lower entropy-based uncertainty reliably marks the modality that was trained on the true class of a test sample, so choosing that modality as dominant is usually correct; the paper checks this on only 30 randomly selected samples per comparison, and if the rule fails on many samples the fusion step has no fallback.","fun_headline_variants_meta":{"raw":{"variants":["Fusing audio and video when each knows different classes","Class-similarity fusion learns from mismatched label sets","One training run recognizes all classes from hetero-labeled modalities","Uncertainty-driven fusion beats baselines in heterogeneous label sets","Shared semantic space bridges modalities with distinct categories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1578,"prompt_tokens":896,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":604}},"tokens_in":512,"tokens_out":682,"duration_ms":7723,"temperature":1.0,"reasoning_tokens":604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:41:44.796748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set with at least a few hundred samples per class-side, compute fused top-1 accuracy, and re-compute it after forcing the higher-uncertainty modality to be dominant; if the forced-wrong choice does not lower accuracy substantially on classes where the entropy rule guessed wrong, the uncertainty claim is empty. More sharply: on samples whose true class was seen only by the modality with higher $u$, the fused accuracy should beat that modality alone; if it does not, the rule is falsified.","supporting_citations":[{"cited_title":"Sophia Koepke, and Zeynep Akata","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP/CLAP backbone features used for all three datasets and the CLIP text encoder used for semantic embeddings."},{"cited_title":"Recent advances in open set recognition: A survey","cited_arxiv_id":null,"evidence_quote":"Motivates the open-set recognition view that maps features into a semantic space to reach unseen classes."},{"cited_title":"Ensemble learning: A survey","cited_arxiv_id":null,"evidence_quote":"Motivates the ensemble of OSRS sub-modules whose disagreement feeds the uncertainty estimate."},{"cited_title":"Deconstructed generation-based zero-shot model","cited_arxiv_id":null,"evidence_quote":"Baseline generalized zero-shot method that the paper's method must beat in the MMHCL comparison."},{"cited_title":"Trusted multi-view classification with dynamic evidential fusion","cited_arxiv_id":null,"evidence_quote":"Baseline trusted multi-view classifier representing the multimodal learning assumption of complete category coverage."},{"cited_title":"Uvat: Uncertainty incorporated view-aware transformer for robust multi-view classification","cited_arxiv_id":null,"evidence_quote":"Baseline incomplete multi-view method that the paper improves on by the stated margins on ActivityNet."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline zero-shot logit adjustment method that the proposed fusion is compared against."}],"review_version":1}