{"id":"5b160bc8-88d5-402c-95af-041275d78af7","arxiv_id":"2507.14935","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors propose OSCMG, an open-set version of Cross Modal Generalization, and show their MICU method with masked contrastive learning and unified jigsaw puzzles outperforms prior methods.","lead":"This paper introduces a new evaluation task, Open-set Cross Modal Generalization (OSCMG), which asks a model pretrained on paired multimodal data to transfer to a new modality while also detecting categories never seen during fine-tuning. It proposes MICU, a method that combines masked fine- and coarse-grained contrastive learning with a modality-agnostic jigsaw puzzle, and reports state-of-the-art results on OSCMG and the existing CMG benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OSCMG benchmark is uninterpretable as reported because no decision rule for rejecting unknown classes is specified; UNK accuracy and HOS cannot be reproduced or compared, so the central SOTA claim is unsupported without this detail.","rationale":"The reader's weakest assumption—that UNK accuracy is undefined—is exactly the load-bearing issue. The entire OSCMG contribution is a benchmark whose headline metric HOS combines known-class accuracy OS* and unknown-class detection UNK. Without a specified reject rule, the UNK column in Table 2 could have been produced in many ways, and comparisons with baselines are not trustworthy. This is not a disagreement with the field's consensus but a missing operational detail. The ablation anomaly in Table 7 strengthens the concern that open-set numbers are not consistently generated. I still credit the paper for releasing code and for consistent gains across many CMG settings; however, the SOTA claim on OSCMG cannot be verified until the decision rule is disclosed and the numbers recomputed. Therefore the reader's conditional verdict remains appropriate; no change is needed.","tokens_in":18266,"tokens_out":4939,"duration_ms":52806,"concrete_test":"Inspect the released code at https://github.com/haihuangcode/CMG to locate the open-set inference path and identify how UNK is computed; then re-run Table 2 using one fixed reject rule (e.g., threshold on maximum softmax probability tuned on a validation split of known classes) for MICU and all baselines. If the released code has no reject rule or the recomputed HOS values change materially, the SOTA claim is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—state-of-the-art HOS on OSCMG—depends entirely on UNK accuracy, the fraction of unseen-class samples correctly rejected. The paper defines OS*, UNK, and HOS in Section 4.1, but never states the decision rule that turns a linear probe trained only on known classes into a detector of unknown classes. Open-set recognition requires an explicit reject mechanism, such as a threshold on maximum softmax probability, an energy score, a distance to prototype, or an extra background class. No such mechanism is described in Sections 3–4, and none is specified for the baselines in Table 2. If different methods use different rules, or if the rule is tuned per method, the HOS comparisons are not apples-to-apples; if no rule exists, the UNK column cannot be reproduced at all. This is load-bearing because the entire OSCMG benchmark, and the claimed SOTA, rests on HOS. The issue is compounded by an apparent arithmetic inconsistency in the ablation Table 7: the row '✓ - ✓' for AVE Split1 V→A reports OS*=7.17, UNK=12.23, HOS=41.34, but computing 2*7.17*12.23/(7.17+12.23) gives approximately 9.04, not 41.34. This suggests the reported open-set numbers may not all be generated by a single well-defined formula. The paper should specify the exact inference-time decision rule and threshold-selection protocol; without it, the experiment cannot be independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Open-set Cross Modal Generalization (OSCMG), a new evaluation protocol that extends the existing Cross Modal Generalization (CMG) task to settings where the target modality contains unknown classes. To address this task, the authors propose MICU, which combines Fine-Coarse Masked multimodal InfoNCE (FCMI) for fine- and coarse-grained cross-modal alignment with Cross-modal Unified Jigsaw Puzzles (CUJP), a modality-agnostic self-supervised task over a shared discrete codebook. The paper reports state-of-the-art results on the new OSCMG benchmark and on the established CMG benchmark, with HOS, precision, F1, and retrieval metrics, alongside ablations and supplementary analyses.","tokens_in":18536,"tokens_out":9963,"duration_ms":90933,"significance":"The paper addresses a timely and relevant problem: how to evaluate and improve multimodal unified representations when downstream test classes are not fully known during training. The proposed task is a natural extension of the authors' prior CMG work and the benchmark construction is reasonable. The methodological components, especially the combination of masked contrastive learning with unified jigsaw puzzles, are interesting and the reported margins over the controlled DCID baseline are nontrivial. The paper also provides code and detailed supplementary analyses. However, the central SOTA claim currently rests on an incompletely specified open-set evaluation protocol, and at least one ablation number is arithmetically inconsistent. These issues must be resolved before the results can be fully trusted.","major_comments":[{"comment":"The open-set decision rule is never specified. The paper defines OS*, UNK, and HOS but does not describe how the linear probe trained only on known classes decides whether a test sample is unknown. No threshold on maximum softmax probability, energy score, distance to prototype, or additional background class is mentioned. This is load-bearing because every UNK value in Tables 2 and 7 depends on this rule, and comparisons across methods are only meaningful if the rule is identical for all methods and is not tuned per method. Please specify the inference-time decision rule, the threshold-selection protocol (e.g., whether a held-out validation set is used), and confirm that the same rule is applied to all baselines.","section":"Section 4.1"},{"comment":"The row '✓ - ✓' for AVE Split1 V→A reports OS*=7.17, UNK=12.23, HOS=41.34. Using the paper's formula HOS = 2 * OS* * UNK / (OS* + UNK), the expected HOS is approximately 9.04, not 41.34. This is a concrete arithmetic inconsistency in a central ablation table. Please correct the value and audit all entries in Tables 7 and 8 for consistency with the HOS formula.","section":"Table 7"},{"comment":"The formal definition of CUJP is difficult to parse. The notation for the permuted segment sequence appears corrupted, and the paper does not specify how the permutation labels are generated, what P is (a hyperparameter or a fixed number), or how the random selection across modalities is implemented. Since CUJP is one of the two main contributions, a precise and unambiguous description is required for reproducibility.","section":"Section 3.3"},{"comment":"The pretraining uses 'text provided by [51]', i.e., the text from the DCID paper, and DCID is also the strongest baseline. Please clarify whether the same pretraining text is used for all baselines and whether the downstream class splits are exactly the same for all methods. This is necessary to rule out an indirect advantage to MICU from shared pretraining data.","section":"Section 4.1 / Table 4"}],"minor_comments":[{"comment":"The superscript notation in the permutation example is garbled; please reformat the equation for readability.","section":"Section 3.3"},{"comment":"The captions refer to 'MICA' instead of 'MICU'; please correct the name.","section":"Supplementary Figure 5"},{"comment":"The rows for DCID and MICU in the UCF(vf)-VGG(a) block contain concatenated numbers (e.g., '69.8762.4974.23 65.01'); please ensure all entries are separated clearly.","section":"Table 5"},{"comment":"The provided code link (https://github.com/haihuangcode/CMG) points to a repository named 'CMG', which may correspond to the prior work rather than this paper; please update to the actual repository for MICU.","section":"Abstract"},{"comment":"The sentence 'The 5th row shows the same effect as the 2nd row' is ambiguous; given the arithmetic issue in row 5, the statement should be clarified and verified.","section":"Section 4.2"},{"comment":"The mask generation mechanism (e.g., whether masks are random per sample or fixed) is not described; please add a sentence specifying how the binary masks are sampled.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable extension of the authors' prior CMG work and the new task is potentially useful. The main concern is the underspecified open-set evaluation protocol, which is fixable. The editor may wish to verify that the GitHub link points to the code for this specific paper, as the current link name suggests it might be for a different project."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: the paper proposes a genuinely useful extension of CMG to open-set conditions, and the method (MICU) is a reasonable combination of masked contrastive learning and modality-agnostic jigsaw puzzles. The experiments are broad and consistently positive. But the evaluation protocol has a hole you need to know about: the paper reports UNK accuracy and HOS without ever stating how unknown-class samples are detected. That makes the central SOTA claim unsupported as written.\n\nWhat's new: the OSCMG task is a natural and worthwhile benchmark—closed-set CMG misses the real-world case where the target modality contains categories the source never saw. The method combines fine/coarse masked InfoNCE with a jigsaw puzzle on quantized features, which is a sensible way to push modality-agnostic representation learning. Compared to previous work, the paper also shows the usual breadth: three datasets, multiple splits, directions, and even optical flow. The code is linked.\n\nWhere it's soft: the open-set decision rule is load-bearing and absent. Section 4.1 defines OS*, UNK, and HOS, but never specifies how a linear probe trained on known classes becomes a detector of unknown classes. No threshold, no energy score, no distance-to-prototype, no background class. If different baselines use different rules, the HOS numbers aren't comparable; if the rule is tuned per method, that's even worse. This is fixable but essential. The table inconsistency in the ablation (AVE Split1 V→A, row '✓ - ✓': OS*=7.17, UNK=12.23, HOS=41.34) doesn't match the harmonic mean formula, which looks like a typo but undermines trust in the open-set numbers. There are also no error bars, and hyperparameters like codebook size are chosen on downstream performance without a clear validation split.\n\nThe task and method are worth engaging with; the paper deserves a serious referee, but it should not be accepted until the authors specify the exact inference-time decision rule and threshold-selection protocol, and ideally release the class splits and code so anyone can reproduce the UNK column. The ablation arithmetic should also be corrected.\n\nThis is a paper for people building multimodal unified representations or evaluating cross-modal transfer in open-set settings. I'd bring it to a reading group, but I'd want to know the decision rule first.\n\nRecommendation: send to peer review with major revision required.","headline":"Useful new task, plausible method, but the missing open-set decision rule makes the headline HOS numbers unreproducible.","tokens_in":19149,"tokens_out":2481,"would_cite":false,"duration_ms":23425,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MICU's masked contrastive learning and modality-agnostic jigsaw puzzles let one multimodal representation handle unseen classes.","keywords":["open-set recognition","cross-modal generalization","multimodal unified representation","masked contrastive learning","self-supervised learning","jigsaw puzzle","vector quantization","audio-visual learning"],"falsifier":"Re-run the OSCMG evaluations with one explicit, fixed unknown-detection rule, for example a threshold on the maximum class score or an energy score, applied identically to every model, and check whether MICU still leads on HOS.","tokens_in":18006,"feed_emoji":"🧩","tokens_out":4739,"duration_ms":48004,"temperature":0.7,"pith_summary":"The paper introduces Open-set Cross Modal Generalization (OSCMG), a task in which a model pretrained on paired multimodal data must transfer from one modality to another while also recognizing classes never seen during downstream training. It argues that prior cross-modal generalization benchmarks are closed-set and therefore miss a common real-world failure mode. To address this, it proposes MICU, which adds masked contrastive learning at both whole-sample and per-timestep levels plus a modality-agnostic jigsaw-puzzle task on discretized features. The paper reports that MICU outperforms prior models on the new OSCMG benchmark and on the established CMG benchmark.","feed_headline":"Open-set test: models must spot unseen classes across modalities","feed_subtitle":"New OSCMG benchmark and MICU method claim top marks while rejecting unknown classes in new modalities.","key_machinery":"The load-bearing machinery is a pair of training signals on a shared discrete codebook: FCMI, an InfoNCE loss applied at coarse and fine temporal scales with sample-specific masking aligned across modalities, and CUJP, a permutation-recognition task over randomly selected quantized segments drawn from any modality. The codebook maps features from all modalities into a common discrete space, and the jigsaw task forces the model to use block-level structure without relying on which modality a block came from. Ablations in the paper show that the coarse contrastive loss is the foundation, while the fine-grained contrastive loss and the jigsaw loss each improve open-set performance.","core_discovery":"The central claim is that a multimodal unified representation can be made open-set-ready by combining two complementary self-supervised objectives. Fine-Coarse Masked Multimodal InfoNCE (FCMI) aligns modalities at two scales, whole-sequence semantics and individual timesteps, with masking applied at aligned positions across modalities. Cross-modal Unified Jigsaw Puzzles (CUJP) quantizes features through a shared codebook, then shuffles randomly chosen segments across modalities and asks the model to identify the permutation, treating all modalities as one. The paper argues this builds a representation that transfers across modalities and lets a linear probe trained on known classes separate known from unknown classes. It reports higher HOS than previous models in 11 of 12 OSCMG settings and consistent gains on the closed-set CMG tasks.","pith_inferences":["The aligned masking in FCMI likely acts as a regularizer that removes modality-specific timesteps, forcing the contrastive loss to rely on shared semantic content, but the paper does not isolate that mechanism directly.","CUJP's benefits may extend beyond video, audio, and optical flow: any paired modalities that can be quantized into segments, such as text spans or skeleton sequences, could use the same permutation task.","Because the paper never specifies the unknown-detection rule behind the UNK accuracy, the reported HOS comparisons across different models should be read as conditional on an unstated evaluation protocol."],"forward_implications":["If OSCMG becomes a standard benchmark, cross-modal models will be judged on whether they can reject unseen classes, not just on known-class accuracy.","Downstream systems for rare modalities such as optical flow could inherit audio and video pretraining with a linear probe and still detect inputs that do not belong to the training classes.","The modality-agnostic jigsaw design cuts the permutation search from factorial in all modalities combined to factorial in a single modality's segment count, making open-set pretraining with three or more modalities practical.","MICU's reported gains on both open-set and closed-set tasks suggest that the added objectives do not trade away ordinary cross-modal alignment quality."],"supporting_citations":[{"why":"Defines the CMG task, supplies the text data used in pretraining, and provides the strongest prior baseline that MICU must beat.","marker":"[51]"},{"why":"Baseline for discrete cross-modal representation learning and the contrastive approach that FCMI extends.","marker":"[29]"},{"why":"Baseline using an optimal-transport codebook for multimodal alignment, compared on OSCMG and CMG.","marker":"[17]"},{"why":"Baseline for multimodal interchange and self-cross-reconstruction, compared across all tasks.","marker":"[58]"},{"why":"The self-supervised jigsaw-puzzle baseline whose modality-sensitive design CUJP is intended to fix.","marker":"[14]"},{"why":"Inspiration for using masking in contrastive learning to build class embeddings that recognize unknown categories.","marker":"[53]"},{"why":"Inspiration for masked contrastive learning that sharpens the boundary between known and anomalous classes.","marker":"[39]"},{"why":"Supplies the VGGSound audio-visual dataset used for unsupervised pretraining of the unified representation.","marker":"[5]"}],"fun_headline_variants":["Open-set cross-modal task: models must handle unseen classes","MICU method: unified representation for open-set cross-modal transfer","Spot unseen classes across modalities with MICU","New OSCMG benchmark: test open-set cross-modal generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported numbers assume that every model being compared decides which inputs are 'unknown' in the same way, but the paper never states what rule the models used to make that decision.","fun_headline_variants_meta":{"raw":{"variants":["Open-set cross-modal task: models must handle unseen classes","MICU method: unified representation for open-set cross-modal transfer","Spot unseen classes across modalities with MICU","New OSCMG benchmark: test open-set cross-modal generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1345,"prompt_tokens":899,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":515,"tokens_out":446,"duration_ms":5114,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:44:16.950655+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the OSCMG evaluations with one explicit, fixed unknown-detection rule, for example a threshold on the maximum class score or an energy score, applied identically to every model, and check whether MICU still leads on HOS.","supporting_citations":[{"cited_title":"Class semantics modulation for open-set in- stance segmentation","cited_arxiv_id":null,"evidence_quote":"Inspiration for using masking in contrastive learning to build class embeddings that recognize unknown categories."},{"cited_title":"Achiev- ing cross modal generalization with multimodal unified rep- resentation","cited_arxiv_id":null,"evidence_quote":"Defines the CMG task, supplies the text data used in pretraining, and provides the strongest prior baseline that MICU must beat."},{"cited_title":"Multi-modal align- ment using representation codebook","cited_arxiv_id":null,"evidence_quote":"Baseline using an optimal-transport codebook for multimodal alignment, compared on OSCMG and CMG."},{"cited_title":"Towards effective multi-modal interchanges in zero-resource sounding object localization","cited_arxiv_id":null,"evidence_quote":"Baseline for multimodal interchange and self-cross-reconstruction, compared across all tasks."},{"cited_title":"Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision","cited_arxiv_id":"2407.01518","evidence_quote":"The self-supervised jigsaw-puzzle baseline whose modality-sensitive design CUJP is intended to fix."},{"cited_title":"Mask2anomaly: Mask transformer for uni- versal open-set segmentation","cited_arxiv_id":null,"evidence_quote":"Inspiration for masked contrastive learning that sharpens the boundary between known and anomalous classes."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the VGGSound audio-visual dataset used for unsupervised pretraining of the unified representation."}],"review_version":1}