{"id":"65d83505-aa22-4bcd-9013-8dcad77ee9ab","arxiv_id":"2501.01470","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"BSS orders multi-modal training samples from balanced to imbalanced using prediction similarity and loss, and reports accuracy gains on six datasets.","lead":"This paper proposes a training-order method for multi-modal models that feeds samples from balanced to imbalanced, where balance is measured by how much the modalities agree and how well the model predicts. The authors report consistent accuracy gains over recent multi-modal rebalancing baselines on six public datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 computes the Eq. 6 balance score from a model whose initialization is unspecified; if θ0 is random, the fixed curriculum is arbitrary, so the claimed mechanism is not established.","rationale":"The reader's weakest assumption identifies exactly the same soft spot: Algorithm 1 scores all samples before any training, and the paper does not establish that those scores are informative. My reading of the manuscript confirms that no warm-up, pretraining statement, or stability analysis is provided for the initial ranking. This is the most load-bearing point because every claimed benefit of BSS is mediated by the ordering induced by Eq. 6. If that ordering is noise, the method's novelty reduces to a fixed random curriculum, and the SOTA results would have to be explained by other differences (e.g., uni-modal losses, late-fusion inference, or seed luck). The proposed permutation test would settle whether the ordering matters; until such a test is provided, the central causal claim is unsupported as written. The verdict of REJECT, even at low confidence, is therefore appropriate, and I do not recommend changing it.","tokens_in":12613,"tokens_out":7798,"duration_ms":80876,"concrete_test":"Run a permutation test on BSS-H: keep all hyperparameters, training loss, and inference unchanged, but replace the sorted x_rank in Algorithm 1 with a random permutation of the same balance scores (or with scores computed from a separately initialized model). Compare test ACC/MAP on CREMA-D and Kinetics-Sounds over at least 5 seeds. Also compute the rank correlation between the initial balance scores and scores recomputed from the same model at epoch Tgrow. If random-order accuracy overlaps BSS-H, or if the initial-to-final rank correlation is near zero, the balance-aware sequence itself is not the driver of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: training from balanced to imbalanced samples, as ranked by the Eq. 6 balance score, drives the reported gains. But Algorithm 1 computes that ranking once before training, after only 'INIT initialize parameters θ0'. The manuscript never states whether θ0 is random, ImageNet/BERT pretrained, or warmed up. This matters because Eq. 6 uses uni-modal softmax predictions and cross-entropy losses. For a randomly initialized network, softmax outputs are near-uniform for every sample, so the prediction-similarity term in Eq. 4 is approximately 1 for all samples and the loss term in Eq. 5 is approximately log(c) for all samples. The resulting balance scores differ only by initialization noise, making the heuristic scheduler's fixed sequence effectively arbitrary and not 'balance-aware' in any data-dependent sense. The paper's Figure 1 toy experiment does not resolve this: it evaluates rankings but does not show whether those rankings come from an untrained model. The learning-based scheduler re-scores every E epochs, but BSS-H relies entirely on the one-time pre-training ranking, and BSS-L's initial sampling probabilities also depend on the same untrusted first score. If the initial rankings are noise, the reported improvements cannot be attributed to the proposed curriculum mechanism without additional evidence. If the authors used pretrained backbones, that assumption must be stated explicitly and ablated, because the entire method hinges on the informativeness of this pre-training evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Balance-aware Sequence Sampling (BSS) for multimodal learning, defined by a balance score that combines uni-modal prediction similarity and training loss (Equation 6). The method orders training samples from balanced to imbalanced and supplies them through either a heuristic curriculum scheduler (BSS-H) or a learning-based probabilistic scheduler (BSS-L) that updates scores every E epochs. Experiments on CREMA-D, Kinetics-Sounds, VGGSound, Twitter2015, Sarcasm, and NVGesture report accuracy/F1/MAP improvements over vanilla fusion and state-of-the-art multimodal rebalancing methods.","tokens_in":12929,"tokens_out":4345,"duration_ms":44493,"significance":"If the central claim holds, the paper identifies a previously underexploited dimension in multimodal imbalance research: the order in which balanced vs imbalanced samples are presented during training. The method is simple, model-agnostic, and tested across audio-video, text-image, and tri-modal datasets, which is a genuine strength. However, the manuscript provides no code, no supplement (despite referencing one), and no error bars, and a key technical detail about model initialization is missing. The contribution is potentially useful, but the current evidence does not yet establish the proposed causal mechanism.","major_comments":[{"comment":"The manuscript never states whether the initial parameters θ0 used to compute the Equation 6 balance scores are randomly initialized, pretrained, or warmed up. Since Equation 6 depends on uni-modal softmax predictions and cross-entropy losses, a randomly initialized classifier will produce near-uniform predictions and near-constant losses across samples, making the one-time ranking X_rank essentially arbitrary. This is load-bearing because BSS-H relies entirely on this pre-training ranking, and BSS-L’s initial sampling probabilities also depend on it. Please specify θ0 precisely and add ablations that vary initialization (random, pretrained, and pretrained with a warmed-up classifier) to demonstrate that the reported gains are attributable to the balance score rather than to initialization noise.","section":"Algorithm 1, Section 3.3"},{"comment":"The normalization operation Norm(·) in Equation 6 is never defined in the main text, and the sentence “More definitions of the criteria are presented in the supplementary materials” cannot be checked because no supplement is included with the submission. The balance score is computed as a difference between a similarity term bounded in [0,1] and a cross-entropy loss that is unbounded; without an explicit normalization, the relative ranking of samples is not well-defined. Please define Norm(·) in the main text and provide the supplementary material in the revision.","section":"Section 3.2, Equation 6"},{"comment":"All experimental results are reported from single runs with no error bars or significance tests, and several margins over the strongest baseline are very small (e.g., Sarcasm ACC 84.41 for BSS-H vs 84.37 for ReconBoost in Table 1). Given the number of hyperparameters in BSS (α, β, λ0, T_grow, E), the claim of state-of-the-art performance needs multiple seeds with means and standard deviations to rule out run-to-run noise.","section":"Tables 1 and 2, Section 4.2"},{"comment":"The motivating experiment comparing curriculum learning with anti-curriculum learning is the only direct evidence in the paper that the sample ordering itself, rather than the pacing schedule, is responsible for the gain. The caption does not say how “learning difficulty” is measured in that toy experiment or whether it corresponds to Equation 6. Please add a control experiment in the main framework that keeps the same balanced-to-imbalanced pacing but randomizes the internal order of the ranked samples, and also tests the reverse order, so that the specific balance-aware ordering mechanism can be separated from a generic curriculum effect.","section":"Section 1, Figure 1(c)"}],"minor_comments":[{"comment":"There are several typos in this section: “Disscssion” should be “Discussion”, “liner function” should be “linear function”, and “pugin” should be “plugin”.","section":"Section 3.3 (before Algorithm 1)"},{"comment":"The softmax in Equation 10 has no temperature parameter; after arbitrary normalization of scores, the resulting sampling probabilities may be either too flat or too peaked. Please state how the normalization affects the effective curriculum and whether sampling in Equation 11 is done with or without replacement.","section":"Section 3.3, Equation 10"},{"comment":"The ablation study is reported on only one dataset (Kinetics-Sounds). To support the general claim that both prediction similarity and training loss contribute to the balance score, similar ablations on at least one text-image dataset and one tri-modal dataset would be helpful.","section":"Section 4.3, Table 3"},{"comment":"Hyperparameter sensitivity is shown only for α and β on the CREMA-D dataset; the curriculum parameters λ0, T_grow, and E, which directly control the BSS-H and BSS-L schedulers, are not analyzed. Adding a sensitivity plot for these parameters would strengthen the robustness claims.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is not self-contained as submitted: the referenced supplementary material is absent, and the initialization of θ0 in Algorithm 1 is unspecified. Both issues directly affect the credibility of the central causal claim. If the authors can clarify the initialization, add the required ablation, provide error bars, and include the supplement, the contribution could become publishable. I recommend major revision rather than rejection because the identified problems are addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a mixed bag: the idea is clean, the experiments are broad, and the writing is honest about what it does, but the central mechanism is not actually established as stated. The balance score in Equation 6 combines prediction similarity and training loss, and the learning-based scheduler that updates the scores every E epochs is a reasonable twist on curriculum learning. That part is new. The paper also tests on six datasets, including a tri-modal one, and the reported gains over strong baselines like MLA are substantial. Credit where due: the authors connect modality imbalance to sample ordering in a way I had not seen explicitly, and the toy experiment in Figure 1 at least tries to show that CL helps while anti-CL hurts.\n\nNow the soft spots. Algorithm 1 computes the balance scores before any training, with only \"INIT initialize parameters θ0.\" The manuscript never says whether θ0 is random or pretrained. If the encoders are pretrained but the uni-modal classifiers are randomly initialized, the softmax outputs are near-uniform, so prediction similarity is roughly 1 for every sample and the loss is roughly log(c). That makes the initial ranking essentially noise. The learning-based scheduler updates scores every 5 epochs, so it becomes informative later, but the initial sampling probabilities and all of BSS-H rely on that first untrusted ranking. This is a real hole, not manufactured. The paper needs to state what θ0 is, add a warm-up, and ablate with an intentionally random first ranking.\n\nThe second soft spot is a confound: Equation 5 adds the uni-modal cross-entropy losses to the objective. Baselines like OGM and MLA do not use that loss. So the reported gains could come from the auxiliary loss rather than the sequence. The ablation in Table 3 compares with/without PreSim and Loss, but never with random sampling under the same loss. That one missing cell would settle it.\n\nMinor: no error bars, the supplement is absent, no code release. The single author and early version suggest this is a work in progress. Still, I would send it to peer review. The question it asks is worth asking, and the method is simple enough that a careful reviewer can quickly test whether the effect is real. Desk rejecting it would be a waste. My own verdict would be skeptical acceptance at best, but it deserves a serious referee.","headline":"The core claim about balance-aware ordering is not yet supported because Algorithm 1 ranks samples before training from an unspecified initialization, but the paper has enough merit to warrant a serious referee.","tokens_in":13389,"tokens_out":2397,"would_cite":false,"duration_ms":25614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ordering multimodal training samples from balanced to imbalanced improves multi-modal learning and outperforms existing rebalancing baselines.","keywords":["multi-modal learning","modality imbalance","curriculum learning","sample sequence","balance score","training scheduler","multi-modal classification"],"falsifier":"Train the same model twice with the same scheduler and random seed but with the ranked order reversed (or with rankings produced from two different random initializations); if performance does not drop, or if the two rankings give different gains, the balance-aware order itself is not what drives the improvement.","tokens_in":12439,"feed_emoji":"⚖️","tokens_out":6411,"duration_ms":51324,"temperature":0.7,"pith_summary":"This paper argues that the order in which multimodal training samples are presented matters as much as the optimization objective: training a model on well-balanced samples first and progressively adding imbalanced ones reduces the gap between strong and weak modalities. It defines a balance score for each sample from two criteria — how similar the two modalities' predictions are, and how well the sample is learned according to training loss — and feeds this score into either a fixed curriculum or a dynamically updated probabilistic sampler. Across six audio-video, text-image, and three-modality datasets, the approach outperforms existing rebalancing methods without changing the network architecture. The practical payoff is a plug-in training strategy for any multimodal classifier.","feed_headline":"Sample order from balanced to imbalanced boosts multi-modal accuracy","feed_subtitle":"Training on balanced samples first, imbalanced later beats 11 rebalance baselines on six datasets.","key_machinery":"The load-bearing object is the balance score $s(x_i)$ of Equation 6, defined as the normalized prediction similarity between modalities minus the normalized total training loss. Prediction similarity (Equation 4) tells how much the two unimodal classifiers agree; the loss term (Equation 5) tells whether the sample is actually learned, guarding against the case where both modalities confidently agree on a wrong answer. This score ranks every training sample once before training (heuristic scheduler) or is updated every $E$ epochs and converted into a softmax sampling probability (learning-based scheduler). The root pacing function of Equation 7 decides what fraction of the most balanced samples is exposed at each epoch, implementing the curriculum from balanced to imbalanced.","core_discovery":"The central claim is that multi-modal learning can be improved by replacing random sample shuffling with a balanced-to-imbalanced sequence, and that this alone can set new state-of-the-art accuracy on CREMA-D, Kinetics-Sounds, VGGSound, Twitter2015, Sarcasm, and NVGesture. The paper's toy experiment shows curriculum ordering (balanced first) outperforms vanilla training while anti-curriculum ordering suppresses it, and the full method—ranking samples by the balance score of Equation 6, then scheduling them with a root pacing function or a learned sampling probability—consistently beats eleven multi-modal rebalance baselines. The authors further show the gain is not an artifact of a particular fusion architecture: BSS works with late fusion, on three modalities, and on top of CLIP-pretrained encoders.","pith_inferences":["A natural test the paper does not run: compare BSS against the same sampling probabilities applied in reverse (imbalanced-to-balanced) under identical compute; the toy experiment suggests the ordering itself, not the non-uniform sampling, is the active ingredient.","Because the balance score depends on initial random parameters, the ranking may be partly seed-dependent; if gains vanish under different initializations, the reported margin would shrink — an easily checkable robustness probe.","The same balanced-to-imbalanced principle could be applied to other training signals, such as per-modality gradient norms or confidence, and could be combined with gradient-modulation methods rather than only replacing them.","The heuristic scheduler's pacing hyperparameters ($\\lambda_0$, $T_{grow}$) are the least principled part; a testable extension is an adaptive pace that responds to the current modality gap instead of a fixed root schedule."],"forward_implications":["If the sequence claim holds, any existing multimodal training pipeline can be upgraded by pre-ranking samples with Equation 6; no architecture change or auxiliary module is required.","The learning-based scheduler, which refreshes balance scores every $E$ epochs, yields larger gains than the fixed heuristic scheduler on every dataset reported, so dynamic re-ranking is the stronger form of the method.","The method transfers to three-modality inputs (RGB, optical flow, depth on NVGesture), where several gradient-modulation baselines do not apply.","Using CLIP-pretrained encoders, sequence sampling still improves over the CLIP baseline and over MLA, indicating the benefit is complementary to strong pretrained features."],"supporting_citations":[{"why":"Establishes the modality imbalance problem and provides the OGM gradient-modulation baseline that BSS must beat, plus the training setup used for audio-video datasets.","marker":"[Peng et al., 2022]"},{"why":"Documents the failure mode where multimodal performance cannot beat the best uni-modal counterpart, motivating the rebalancing goal.","marker":"[Wang et al., 2020]"},{"why":"Supplies the curriculum learning framework of ordering samples from easy to hard, which BSS repurposes from balanced to imbalanced.","marker":"[Bengio et al., 2009]"},{"why":"Provides the pacing-function machinery, including the root function used in Equation 7 of the heuristic scheduler.","marker":"[Hacohen and Weinshall, 2019]"},{"why":"Supports using uni-modal predictions as supervision, justifying the prediction-similarity criterion in the balance score.","marker":"[Zou et al., 2023]"},{"why":"Provides the CLIP-pretrained encoders used in the robustness experiments on text-image datasets.","marker":"[Radford et al., 2021]"}],"fun_headline_variants":["Balanced-to-imbalanced sample order boosts multi-modal accuracy","Training order: balanced first, imbalanced later improves multi-modal","New SOTA: sequence sampling from balanced to imbalanced in MML","Balanced-first ordering improves multi-modal learning, beats 11 baselines","Start balanced, end imbalanced: better multi-modal training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The balance ranking is computed once from a randomly initialized model, and the method depends on those initial predictions and losses being informative enough to sort samples from balanced to imbalanced.","fun_headline_variants_meta":{"raw":{"variants":["Balanced-to-imbalanced sample order boosts multi-modal accuracy","Training order: balanced first, imbalanced later improves multi-modal","New SOTA: sequence sampling from balanced to imbalanced in MML","Balanced-first ordering improves multi-modal learning, beats 11 baselines","Start balanced, end imbalanced: better multi-modal training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"total_tokens":2197,"prompt_tokens":884,"completion_tokens":1313,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1235}},"tokens_in":500,"tokens_out":1313,"duration_ms":9981,"temperature":1.0,"reasoning_tokens":1235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:43:14.211368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model twice with the same scheduler and random seed but with the ranked order reversed (or with rankings produced from two different random initializations); if performance does not drop, or if the two rankings give different gains, the balance-aware order itself is not what drives the improvement.","supporting_citations":[{"cited_title":"Balanced multimodal learning via on-the-fly gradient modulation","cited_arxiv_id":null,"evidence_quote":"Establishes the modality imbalance problem and provides the OGM gradient-modulation baseline that BSS must beat, plus the training setup used for audio-video datasets."},{"cited_title":"What makes training multi-modal classification net- works hard? In CVPR, pages 12692–12702,","cited_arxiv_id":null,"evidence_quote":"Documents the failure mode where multimodal performance cannot beat the best uni-modal counterpart, motivating the rebalancing goal."},{"cited_title":"Curriculum learning","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum learning framework of ordering samples from easy to hard, which BSS repurposes from balanced to imbalanced."},{"cited_title":"On the power of curriculum learning in training deep networks","cited_arxiv_id":null,"evidence_quote":"Provides the pacing-function machinery, including the root function used in Equation 7 of the heuristic scheduler."},{"cited_title":"Unis-mmc: Multimodal classification via unimodality- supervised multimodal contrastive learning","cited_arxiv_id":null,"evidence_quote":"Supports using uni-modal predictions as supervision, justifying the prediction-similarity criterion in the balance score."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP-pretrained encoders used in the robustness experiments on text-image datasets."}],"review_version":1}