{"id":"2b450461-019a-4fa4-83aa-0354654d430a","arxiv_id":"2506.11550","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Data Remixing improves multimodal learning by decoupling samples into per-modality subsets and training each batch on a single modality, yielding accuracy gains on CREMAD and Kinetic-Sounds.","lead":"This paper introduces a training strategy called Data Remixing that masks the stronger modality in each multimodal sample and reorganizes batches so each batch trains only one modality. On two audio-visual datasets, the method improves classification accuracy by 3 to 8 percentage points over standard fusion, with no added inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 10's simplification assumes zeroed inputs produce zero encoder features; with ResNet-18's BatchNorm this fails, so the claimed batch-level interference removal is not actually implemented as described.","rationale":"The reader's weakest assumption concerned whether single-modality batches could harm cross-modal learning; my concern is more direct and technical: the method does not actually create single-modality batches in the feature space because zeroed inputs still propagate through BatchNorm in ResNet-18. The paper's formal argument (Eq. 6 to Eq. 10) depends on the masked branch's output being exactly zero, which is not true for the implemented architecture. This is load-bearing because the paper's novelty and method design are justified by the batch-level interference analysis; if the mechanism is invalid, the improvements must be attributed to sample rebalancing, alternating optimization, or other factors. The empirical results may remain valid, but the central explanation requires major revision. Since the reader already issued CONDITIONAL, this concern reinforces that verdict without moving it; hence UNCHANGED.","tokens_in":12264,"tokens_out":12021,"duration_ms":116982,"concrete_test":"In the released code, add a forward/backward hook for a video-only batch (audio input zeroed) and compute (a) the L2 norm of the audio encoder output z_a and (b) the L2 norm of the gradient of the multimodal CE loss with respect to the audio encoder parameters. If either is non-negligible (e.g., >1% of the video branch gradient norm), Eq. 10's premise fails; then re-run Table 5 with explicit feature-level masking (setting z_a=0 before fusion) to see whether the reassemble gain persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanistic claim—that reassembling batches to contain a single modality eliminates cross-modal interference—rests on the simplification in Eq. 10, where masking the other modality to zero removes its contribution to the logits. This requires z_a = 0 for zeroed audio inputs. Under the paper's own architecture (ResNet-18, Section 4.1), this is false: BatchNorm layers apply an affine transformation to zero input using running statistics, so a zeroed input produces a nonzero feature vector; residual connections propagate this. Hence, in a 'video-only' batch, the audio branch still contributes to f(x_i) in Eq. 10 and still receives nonzero gradients (e.g., through BN scale/shift parameters). The claimed removal of cross-modal interference is therefore not achieved as described, and the observed gains cannot be attributed to the stated batch-level mechanism without an additional control. Supporting this concern, the 'Reassemble' ablation in Table 5 does not mask the non-selected modality, so each sample still contains both modalities, meaning Eq. 9 is not actually exercised in that ablation. The empirical results may still hold, but the paper's central explanation and novelty are undermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training strategy called Data Remixing for multimodal models, targeting the two problems of modality laziness (weak modalities learning too slowly) and modality clash (cross-modal gradient interference). The method first decouples multimodal inputs by evaluating per-sample unimodal separability using KL divergence to a uniform distribution, retaining only the weaker modality for each sample. It then reassembles training batches so that each batch contains samples from only one such modality subset, with the non-selected modalities masked to zero. The authors claim this batch-level control eliminates cross-modal interference, and demonstrate on CREMAD and Kinetic-Sounds consistent accuracy improvements over conventional fusion baselines and several prior imbalance-aware methods, without dataset expansion or inference overhead. The paper includes ablations of the two components, analyses of sample allocation and gradient alignment, and comparisons of unimodal prediction strategies.","tokens_in":12487,"tokens_out":7421,"duration_ms":67602,"significance":"If the empirical results hold, the method is attractive for its simplicity, architecture-agnosticism, and consistent gains across two audio-visual benchmarks and multiple fusion architectures (Concat, Summation, Decision Fusion, FiLM, Bi-Gated, MMTM, CentralNet). The authors provide the source code, and the ablations give some support for both design choices. However, the paper's central mechanistic claim—that mixing modalities within a batch causes interference and that single-modality batches with zeroed other modalities remove that interference—is undermined by an incorrect simplification in the derivation of Eq. (10). The claimed batch-level novelty therefore needs either a corrected mechanism with supporting evidence or a revised, more modest framing. The empirical gains are still potentially valuable, but the current explanation does not reliably attribute them to the stated cause.","major_comments":[{"comment":"The simplification of the batch loss to depend only on the retained modality assumes that masking a modality input to zero yields a zero encoder output (z_a = 0). Under the architecture used in Section 4.1 (ResNet-18 encoders), this is false: BatchNorm layers apply an affine transformation to a zero input using running statistics, and residual connections propagate nonzero signals, so a zeroed input still produces a nonzero feature vector. Consequently, the zeroed modality still contributes to f(x_i) in Eq. (5) and still receives nonzero gradients from the batch loss. The claimed elimination of cross-modal interference at the batch level is therefore not actually achieved by the described procedure. Please provide empirical evidence that the contribution of the zeroed branch is negligible for the trained encoders (e.g., gradient norms of the masked branch), or modify the method to explicitly zero the encoder output (or detach the gradient) for the non-selected modality and re-run the experiments. This is load-bearing because the paper's central novelty is the batch-level interference-removal mechanism.","section":"Section 3.2, Eq. (10)"},{"comment":"The 'Reassemble' ablation is described as reassembling batches without modality masking ('without modality masking'). However, the formal definition of the modality subset in Eq. (7) requires that non-selected modalities are zeroed: D_mk = {x_i | x_j_i = 0 for all j ≠ k}. Without masking, each sample still contains both modalities, so the batch does not satisfy the condition in Eq. (9) as formalized. The ablation therefore cannot isolate the effect of batch-level reassembling under the proposed mechanism; the improvement could be due to a change in sampling distribution (e.g., over-sampling the weak modality) rather than to the removal of cross-modal interference via zeroed inputs. Please include an ablation that reassembles batches while masking the non-selected modality (the full Eqs. 7-9 setup) and report the individual contributions of masking and batch composition separately.","section":"Section 4.5.1, Table 5"},{"comment":"The text states 'our approach neither expands the dataset nor reduces the model's training efficiency,' but Table 3 shows that Remix training time to convergence is 2537s on CREMAD and 4946s on Kinetic-Sounds, compared to 1536s and 3849s for the baseline—an increase of about 65% and 29%, respectively. This directly contradicts the stated claim about training efficiency. Please revise the claim to state that the method does not expand the dataset, that it is more efficient than Resample and MLA, and acknowledge a non-negligible increase in training time relative to the plain baseline.","section":"Section 4.3, Table 3"},{"comment":"The headline numbers 'approximately 6.50% up on CREMAD and 3.41% up on Kinetic-Sounds' do not match any comparison reported in Table 2. The improvement of Remix over the Concatenation baseline is 8.20% and 5.40%; the improvement of MLA+Remix over MLA is 6.18% and 3.09%; Resample+Remix over Resample is 5.64% and 3.23%. Please clarify the reference baseline for the claimed 6.50% and 3.41% figures, or correct the numbers and the associated text.","section":"Abstract and Section 1"}],"minor_comments":[{"comment":"The weights w_k in the auxiliary losses are not specified; please state how they are chosen in the experiments (e.g., all equal to 1) and whether they are tuned per dataset.","section":"Section 3.1, Eq. (1)"},{"comment":"The paper reports averages over three random seeds but no standard deviations or confidence intervals; please include variance information in the tables or at least in an appendix.","section":"Section 4.1"},{"comment":"The 'Decision fusion' method (Gunes & Piccardi, 2005) is not defined; please specify the exact aggregation rule (e.g., averaging softmax probabilities) to ensure reproducibility.","section":"Section 4.2"},{"comment":"The reported unimodal accuracy improvements (audio +1.75%, video +2.96% on CREMAD) are not tied to a clearly defined baseline; please specify whether these are relative to the baseline model's unimodal classification heads and at which training stage they are measured.","section":"Section 4.5.3"},{"comment":"The 'ideal guidance direction' used to compute gradient alignment is not defined; please specify how it is computed and which gradient (e.g., of the audio branch) is being measured.","section":"Figure 3(c)"},{"comment":"The warm-up epoch count E_r is a free parameter; the text mentions a 10-epoch warm-up, but please state explicitly whether the same value is used for both datasets and whether it was tuned.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the incorrect simplification in Eq. (10), which undermines the paper's central mechanistic claim. If the authors can supply empirical evidence that the zeroed branch's contribution is negligible (or modify the method to truly zero it), the paper could become acceptable. The efficiency contradiction in Table 3 is also problematic and needs correction. The empirical results appear consistent, but the over-claim in the abstract about 'first to analyze at batch level' should be tempered unless the mechanism is convincingly established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Data Remixing is a simple, plausible training recipe that gives consistent accuracy gains across fusion methods on two audio-visual datasets, but the paper's mechanistic explanation is shakier than the results.\n\nWhat's new: the combination of sample-level KL-divergence selection of the weaker modality and then re-batching so each batch is single-modality is, to my knowledge, new. The method is architecture-agnostic, doesn't expand the training set, and adds no inference cost. The experiments are reasonably thorough: ablations of both components, combinations with existing methods like Resample and MLA, and a couple of complex fusion architectures. The training time comparison in Table 3 is a nice touch.\n\nSoft spots: the biggest one is theoretical. Eq. 10 simplifies the batch loss by assuming that masking a modality to zero removes its contribution to the f(x_i) logits. With ResNet-18 and BatchNorm, a zeroed input does not produce a zero feature; the affine transform from running statistics means the masked branch still contributes and receives gradients. So the claimed removal of cross-modal interference isn't exactly what the code does, and the derivation doesn't justify the mechanism as written. The 'Reassemble' ablation in Table 5 doesn't mask the non-selected modality, so it only tests batch composition, not the full interference-removal story. Also, the baseline 'Concatenation' in Tables 1, 2, and 5 doesn't appear to include the auxiliary unimodal classification heads added via Eq. 1, so part of the gain could be from those heads rather than the remixing. Table 6 shows the gain persists with different unimodal prediction methods, but it's not a clean control. Finally, no error bars or significance tests, despite three seeds.\n\nThe novelty claim about being 'first to analyze the challenges at the batch level' is overstated; the ingredients exist in prior work, though the specific combination is new. The circularity burden is low: Eq. 10 is motivational, not evidence, and the accuracy numbers are external measurements.\n\nBottom line: this is a solid empirical contribution that a multimodal learning researcher should know about, but the theory needs to be either fixed or reframed as a heuristic. It deserves a serious referee; I'd send it to review and ask for a revision that controls for the auxiliary heads, reports variance, and reconciles Eq. 10 with the actual BatchNorm behavior.","headline":"A plausible training recipe with consistent empirical gains, but the paper's interference-removal story doesn't hold up under scrutiny.","tokens_in":13016,"tokens_out":4032,"would_cite":true,"duration_ms":38029,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Batch-level interference, not just slow modalities, causes weak multimodal learning; Data Remixing trains each batch on one modality and lifts accuracy by up to 6.5 percentage points.","keywords":["multimodal learning","modality laziness","modality clash","data remixing","batch-level interference","unimodal sufficiency","audiovisual classification","gradient alignment"],"falsifier":"Measure, on a held-out dataset, the angle between a batch's gradient and the gradient of the same samples with only one modality active; the paper predicts this discrepancy is smaller after reassembling. If mixed-modality batches show no larger discrepancy than remixed batches, or if single-modality batches reduce accuracy on a task where cross-modal complementarity is essential, the batch-level story is incomplete.","tokens_in":12054,"feed_emoji":"🎬","tokens_out":11397,"duration_ms":97100,"temperature":0.7,"pith_summary":"This paper argues that two failures of multimodal training—modality laziness, where the strong modality dominates and the weak one learns too little, and modality clash, where inconsistent gradient directions from different modalities interfere—can both be traced to how data is grouped into batches, and that fixing the grouping fixes both. The proposed Data Remixing strategy first decouples each training sample so only the weaker modality remains, allocating more and harder samples to the weaker modality, then reassembles samples so that every batch contains only one modality. On the CREMAD and Kinetic-Sounds benchmarks, adding this training strategy to existing fusion methods improves accuracy by roughly 6.50 and 3.41 percentage points, respectively, without expanding the dataset or adding inference-time cost. The paper claims this is the first batch-level analysis and solution for these problems. In the conclusion, the authors note a limitation: when one modality is auxiliary and carries limited information, the unimodal evaluation and allocation strategy may require refinement.","feed_headline":"Single-modality batches lift multimodal accuracy by up to 6.5%","feed_subtitle":"A simple data-remixing schedule fixes both lazy and clashing modality failures on CREMAD and Kinetic-Sounds.","key_machinery":"The machinery is a two-stage data-remixing schedule. Stage one is a sample-level unimodal capability score: for each sample, compute the KL divergence $D_{\\mathrm{KL}}(p^k_i \\| U)$ between each modality's prediction and the uniform distribution; the modality with the smaller value is judged less separable, i.e. weaker on that sample, and is the only modality kept. Stage two is a batch-composition constraint: after partitioning the data into subsets $\\mathcal{D}_{m_k}$ by retained modality, every mini-batch is drawn entirely from one subset, so the batch gradient mixes gradients of only one modality. The paper argues that this algebraically removes the cross-modal interference terms from the batch cross-entropy loss, making each modality's update direction consistent with a unimodal objective.","core_discovery":"The paper's central claim is that insufficient and imbalanced multimodal learning has a batch-level cause: even after decoupling multimodal data into unimodal samples, a batch containing both modalities still produces gradient updates whose directions interfere, so each modality deviates from its ideal optimization path. Data Remixing therefore consists of two linked operations: decoupling, in which each sample is assigned to the modality whose unimodal prediction has the smaller KL divergence to the uniform distribution (the weaker modality), with other modalities masked to zero; and reassembling, in which the decoupled subsets are sampled so that each mini-batch $B_i$ satisfies $B_i \\subseteq \\mathcal{D}_{m_k}$ for one modality $k$. The paper shows that with such batches, the cross-entropy loss no longer mixes $W^{a}z^{a}$ and $W^{v}z^{v}$ terms for different modalities within a batch, which it identifies as the source of modality clash. After a warm-up stage on the full multimodal data, training alternates between modality-specific batches, exposing the weak modality to more samples and aligning each modality's gradient direction, yielding the reported gains.","pith_inferences":["Extension: if batch-level interference is the true bottleneck, the same remixing schedule should transfer to other paired-modality objectives such as audiovisual retrieval or video captioning, where batch composition is equally controllable.","Extension: the KL-to-uniform score is a cheap proxy for sample-level weakness; a testable variant would use per-sample holdout validation or true-label confidence to assign samples, and check whether the accuracy gains change.","Extension: the mechanism predicts a measurable signature—the cosine angle between a batch update and a pure unimodal update should be consistently smaller for remixed batches; tracking this for both modalities, not just the strong one, would sharpen the evidence.","Extension: the paper's stated auxiliary-modality limitation suggests a boundary: when one channel carries almost no task information, over-allocating training to it could waste capacity, and the allocation rule may need a floor on how many samples each modality receives."],"forward_implications":["Data Remixing can be layered on top of existing balance methods: combining it with MLA gives 74.19% on CREMAD and with Resample gives 58.40% on Kinetic-Sounds, beating either method alone.","The strategy generalizes across fusion choices—concatenation, summation, decision fusion—and across intermediate-fusion architectures such as MMTM and CentralNet, so it is not tied to one model design.","Because the training set is not expanded, Data Remixing converges faster in wall-clock time than Resample and MLA under the same hardware and batch size.","Reassembling single-modality batches aligns each modality's gradient direction with its ideal direction, which the paper demonstrates by measuring gradient-angle discrepancies during training on CREMAD."],"supporting_citations":[{"why":"Provides the Gradient-Blending baseline, the per-modality classification-head loss used in warm-up, and the observation that optimization speeds differ across modalities.","marker":"(Wang et al., 2020)"},{"why":"Supplies the OGM-GE gradient-modulation baseline and the imbalance ratio $\\rho$ used to show decoupling balances modality learning.","marker":"(Peng et al., 2022)"},{"why":"Supplies the Resample sample-level Shapley-valued baseline that remixing builds on and improves, and the dropout unimodal-prediction alternative.","marker":"(Wei et al., 2024)"},{"why":"Supplies the MLA alternating-unimodal baseline that remixing is combined with and improves.","marker":"(Zhang et al., 2024b)"},{"why":"Supplies the PMR prototypical-rebalance baseline and the cited evidence that gradient update directions differ across modalities.","marker":"(Fan et al., 2023)"},{"why":"Supplies the unimodal-teacher method and the observation that multimodal models converge on the faster-optimizing modality before the weaker one is learned.","marker":"(Du et al., 2021)"},{"why":"Provides the CREMA-D audiovisual emotion dataset used for the main experiments.","marker":"(Cao et al., 2014)"},{"why":"Provides the Kinetic-Sounds audio-visual action dataset used as the second benchmark.","marker":"(Arandjelovic & Zisserman, 2017)"}],"fun_headline_variants":["Data remixing aligns gradients to fix lazy and clashing modalities","Single-modality batches stop cross-modal interference in training","Remixing data into unimodal batches boosts multimodal accuracy by 6.5%","Decouple, filter, reassemble: a recipe for balanced multimodal learning","Avoiding modality clash with batch-level data remixing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that interference between modalities arises from putting different modalities in the same training batch, and that removing that mixing by training each batch on a single modality improves multimodal learning without sacrificing the cross-modal features that joint training is meant to provide.","fun_headline_variants_meta":{"raw":{"variants":["Data remixing aligns gradients to fix lazy and clashing modalities","Single-modality batches stop cross-modal interference in training","Remixing data into unimodal batches boosts multimodal accuracy by 6.5%","Decouple, filter, reassemble: a recipe for balanced multimodal learning","Avoiding modality clash with batch-level data remixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1519,"prompt_tokens":976,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":592,"tokens_out":543,"duration_ms":5987,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:04:07.091942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a held-out dataset, the angle between a batch's gradient and the gradient of the same samples with only one modality active; the paper predicts this discrepancy is smaller after reassembling. If mixed-modality batches show no larger discrepancy than remixed batches, or if single-modality batches reduce accuracy on a task where cross-modal complementarity is essential, the batch-level story is incomplete.","supporting_citations":[{"cited_title":"What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\\ 12695--12705, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the Gradient-Blending baseline, the per-modality classification-head loss used in warm-up, and the observation that optimization speeds differ across modalities."}],"review_version":1}