{"id":"e4cab68e-9963-4491-bfa5-71265b5d7d99","arxiv_id":"2506.12724","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DMS reweights image and text inputs per sample using confidence, MC-dropout uncertainty, and semantic similarity, and reports improved MLLM accuracy and robustness.","lead":"This paper proposes a scheduling method that gives each input modality, such as image and text, a per-sample weight before combining them in a multimodal model, based on how confident, certain, and aligned each modality is. The authors report accuracy gains on VQA, captioning, and retrieval, especially when inputs are corrupted.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The integration claim is unsupported: BLIP-2 and LLaVA do not fuse via a weighted sum of unimodal encoder embeddings, so Eq. (3.1) cannot be applied 'on top' without an unspecified architectural transformation.","rationale":"The reader's verdict is REJECT with high correctness risk; my stress-test agrees. I considered the Modality Weight Consistency Loss (§3.4, Prop. 4.2) as an alternative primary concern. Because h is defined as the weighted average Σ ω_m f^(m), L_mwcl reduces to a weighted pairwise agreement term Σ_{j<k} ω_j ω_k ||f^(j) - f^(k)||^2; this is a substantive regularizer only if the unimodal encoders are trainable, and the paper never states their trainability. That ambiguity is serious, but the backbone-integration mismatch is more decisive for the headline claim because the experimental tables are the only positive evidence and no forward pass is specified for the claimed backbones. The reader's weakest_assumption names the same gap, and my reading of Sections 3.1, 3.2, 3.6 and the BLIP-2/LLaVA literature confirms it. The paper also contains literal placeholders and no reproducibility artifacts, which strengthens the need for the forward-pass check rather than a more charitable reading. A negative or unanswerable result on that check would leave the central claim unsupported; a positive result, with code showing Eq. (3.1) actually executed on these backbones, would at best make the central claim conditional on further empirical validation. Thus no verdict change from REJECT; the concern reinforces the existing rejection.","tokens_in":7788,"tokens_out":7014,"duration_ms":81564,"concrete_test":"Obtain the exact forward pass used for the 'DMS (Ours)' rows: for BLIP-2, give the source and dimension of f^(image) and f^(text), the definition of p^(image) used in Eq. (3.2), and the operation that consumes h. Then check whether the code literally computes h = Σ ω f and routes h into the decoder, or whether the backbone still performs Q-Former cross-attention or token concatenation. If the latter, reproduce Table 2's BLIP-2 DMS row and Table 4's LLaVA row with an implementation of the literal Eq. (3.1). Unless the reported margins survive that implementation, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the fused representation as h = Σ_m ω_m(x) f^(m)(x^(m)), and Section 3.6 says DMS is applied 'on top of the modality-specific encoder outputs, prior to feeding into the multimodal Transformer or language decoder.' The headline empirical claim is that this improves BLIP-2 and LLaVA (Tables 1–4). Neither backbone computes a convex combination of two unimodal embeddings. BLIP-2 uses a Q-Former whose learned queries cross-attend to image features and are then passed to the LLM; text is handled inside the LLM. LLaVA projects image patch tokens with an MLP and concatenates them with text token embeddings before the LLM. There is no common vector f^(image) and f^(text) whose weighted sum h is the fusion. The three scheduling signals in §3.2 are also undefined for these backbones: confidence and uncertainty require a softmax classifier p^(m) per modality, but an image encoder has no such head and the language side is generative. To instantiate DMS the authors must specify how token sequences are pooled, how p^(m) is computed for images, and how h replaces cross-attention or concatenation; none of this appears. Without that mapping, the 'DMS (Ours)' rows cannot be verified as the method described. This is load-bearing because it is the only bridge between the proposed mechanism and the claimed results. The unresolved 'MMBT [?]' citation in §2.4, the 'Figure??' placeholder in §5.4, and the absence of code, seeds, and hyperparameters are consistent with an incomplete specification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Modality Scheduling (DMS), a per-sample weighting scheme for multimodal fusion. Each modality's embedding is weighted by a softmax over confidence (1 minus predictive entropy), negative Monte-Carlo dropout uncertainty, and semantic alignment (cosine similarity among unimodal embeddings). The authors add a Modality Weight Consistency Loss (MWCL) that penalizes distance between the fused representation and modality embeddings, weighted by the same soft weights. They claim DMS is model-agnostic and can be applied to BLIP-2 and LLaVA, and they report improvements on VQA v2, MSCOCO Captioning, and Flickr30K Retrieval, plus robustness gains under synthetic image/text corruption. A theoretical section gives an approximation bound, a MWCL variance identity, and a Rademacher generalization bound.","tokens_in":8100,"tokens_out":2775,"duration_ms":31971,"significance":"If the central claim were established, DMS would offer a simple, interpretable alternative to static fusion for instance-level robustness, with plausible practical value because it requires no architectural overhaul. However, the manuscript's two load-bearing pieces—the claimed applicability to BLIP-2/LLaVA and the theoretical justification—are not supported as written. The experiments are presented without reproducibility details. The paper does include a falsifiable empirical setup (Tables 1-4), but the missing integration mapping and the vacuous Proposition 4.2 prevent this from being a verifiable contribution. The theoretical section is either trivial or generic and does not validate the method's specifics.","major_comments":[{"comment":"The method, as defined in Eq. (3.1), computes the fused representation as h = Σ_m ω_m f^(m)(x^(m)), a convex combination of unimodal embeddings, and requires per-modality softmax probabilities p^(m) to compute confidence. Neither BLIP-2 nor LLaVA fuses modalities in this way: BLIP-2 uses a Q-Former whose fixed queries cross-attend to image features and then feed the LLM, with text handled inside the LLM; LLaVA projects image patches with an MLP and concatenates the resulting token embeddings with text tokens before the LLM. There is no common f^(image) and f^(text) whose convex combination h is the fusion, and the image encoder has no softmax classification head to supply p^(m). The paper does not specify how token sequences are pooled into f^(m), how p^(m) is computed for images or for the generative language side, or how h replaces cross-attention or concatenation. Without that mapping, the reported 'DMS (Ours)' rows in Tables 1–4 cannot be verified as implementing the described method. This is the central bridge between the proposed mechanism and the claimed results, and it is missing.","section":"§3.1, §3.6, §5.1–§5.6"},{"comment":"Proposition 4.2 states E_m[||h − f^(m)||_2^2] = Var(f^(m)) − ||h − f̄||_2^2, but by the paper's own definition in Eq. (3.1), h = f̄ = Σ_m ω_m f^(m). Therefore h − f̄ = 0 exactly, and the identity reduces to E_m[||h − f^(m)||_2^2] = Var(f^(m)) with no free parameters. The MWCL term is simply the weighted variance of the unimodal embeddings around their weighted mean; it imposes no additional consistency constraint beyond the definition of h. The claimed 'regularization effect' is therefore a restatement of the fusion definition, not a theoretical justification. This is load-bearing for the paper's claim of a novel consistency loss.","section":"§4.3, Proposition 4.2"},{"comment":"Theorem 4.3 is stated without proof, and as written it is a generic Rademacher bound R(f) ≤ R̂(f) + 2L·R_n(F) + sqrt(log(1/δ)/2n) that does not depend on the DMS scheduling signals or on the specific form of ω_m(x). The subsequent claim that 'our dynamic weighting mechanism indirectly controls R_n(F)' is not supported by any calculation, and the bound would apply identically to any hypothesis class over the same feature space. Since the theoretical section is offered as a contribution, the missing proof and the non-specificity of the bound are material.","section":"§4.4, Theorem 4.3"},{"comment":"The experimental section reports no error bars, no number of random seeds, no hyperparameter values for α, β, γ, λ, or T, and no code release. Table 1 and Table 3 report differences as small as 0.6–1.3 points with no indication of variance; Table 4 reports 'Helpfulness (score)' without defining the metric or its scale; and Section 5.4 refers to a non-existent 'Figure??'. As a result, the referee cannot assess whether the reported gains are statistically meaningful or reproducible, and the claimed 2.3% VQA improvement and 3.1% R@1 improvement are not verifiable from the information provided.","section":"§5.1–§5.6"}],"minor_comments":[{"comment":"The citation 'MMBT [?]' is unresolved; the reference list has no corresponding entry. Please either cite the actual MMBT paper or remove the citation.","section":"§2.4"},{"comment":"The text refers to 'Figure??' for the modality weight visualization; the figure is missing or the cross-reference was never filled in.","section":"§5.4"},{"comment":"The LLaVA citation (reference [4]) lists incorrect author names; the authors of 'Visual Instruction Tuning' are Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.","section":"References"},{"comment":"The paper does not state which split of VQA v2 is used (e.g., train/val or test-dev), which split of Flickr30K is used for retrieval evaluation, or the evaluation protocol for captioning (e.g., beam size, length penalty). These details are needed for reproducibility.","section":"§5.1"},{"comment":"The term 'epistemic uncertainty' is used for MC dropout variance, but no distinction is made between aleatoric and epistemic uncertainty; this is fine, but the paper should clarify that u_m is a proxy, not a calibrated uncertainty estimate.","section":"§1 and §3.2"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an unfinished draft: it contains unresolved LaTeX placeholders ('MMBT [?]' and 'Figure??'), garbled author names in a reference, and no code or hyperparameter details. More importantly, the central claim of integration with BLIP-2/LLaVA is unsupported by any architectural mapping, and Proposition 4.2 is vacuous by the paper's own definitions. These are not local issues that a minor revision could fix; they undermine the method as presented and the validity of the empirical tables. I recommend rejection, though I would be open to a substantially rewritten version that provides an explicit, non-trivial adaptation of DMS to actual MLLM architectures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core mechanism—softmax weighting over confidence, MC-dropout uncertainty, and cosine alignment—is a reasonable composition of known tools, and as a generic module for architectures that literally sum unimodal embeddings it could work. Second, the paper as written does not support its headline claims: the consistency loss is a tautology, the theory section contains no proof where one is needed, and the mapping onto BLIP-2/LLaVA is never specified.\n\nWhat is actually new: the specific combination of the three scheduling signals is not in the cited literature, and the ablation study is sensibly structured. If you ignore the claims about BLIP-2/LLaVA, the formulation is clear and the motivation—instance-level reweighting for noisy inputs—is reasonable.\n\nThe soft spots are structural, not cosmetic. Eq. (3.1) defines h as the weighted sum of the f(m), and Eq. (3.4) defines the MWCL as the same weights times squared distances to those f(m). That is the weighted variance of the embeddings around their own weighted mean. It has no adjustable parameters under the stated definition, and Proposition 4.2 merely restates that identity. If the weights were learnable, the trivial minimum would put all mass on one modality. So the regularizer cannot do what the paper claims. Theorem 4.3 is a generic Rademacher bound with no proof and no DMS-specific content; Proposition 4.1 is a simple Lipschitz argument that says little.\n\nThe integration claim is the load-bearing flaw. BLIP-2 fuses via Q-Former cross-attention; LLaVA concatenates projected image tokens with text embeddings before the LLM. Neither produces a common vector f^(image) and f^(text) whose weighted sum is h. The paper never says how token sequences are pooled, how p^(m) is computed for images, or how h replaces cross-attention or concatenation. Without that mapping, the 'DMS (Ours)' rows in Tables 1–4 cannot be verified as the described method. This is not a minor omission; it is the only bridge between the mechanism and the results.\n\nThe experimental reporting is also below the bar: no code, no seeds, no hyperparameters, no error bars, and the text contains literal placeholders (\"Figure??\" in 5.4, \"MMBT [?]\" in 2.4). I am not inclined to trust numbers that cannot be reproduced or even tied to a concrete architecture.\n\nWho gets value out of this? A reader working on adaptive fusion in simple dual-encoder models might find a starting point, but they would have to reimplement everything and fix the loss themselves. The paper does not deserve a serious referee in its current form; I would desk-reject it and invite a resubmission that specifies the architectural mapping, removes or redefines the consistency loss, and provides reproducible experiments.","headline":"Load-bearing integration gap plus a vacuous consistency loss sink an otherwise plausible adaptive-fusion idea.","tokens_in":8646,"tokens_out":2286,"would_cite":false,"duration_ms":27331,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-sample modality weights, set by confidence, uncertainty, and semantic alignment, lift VQA accuracy by 2.3 points and halve noise-induced degradation compared with static fusion.","keywords":["dynamic modality scheduling","multimodal fusion","confidence estimation","Monte Carlo dropout","uncertainty quantification","semantic consistency","visual question answering","multimodal robustness"],"falsifier":"Run the exact pipeline on a standard BLIP-2 checkpoint without removing the Q-Former: if inserting DMS weights before the multimodal transformer does not reproduce roughly 74.4 VQA accuracy and the reduced noise degradation, the claim fails. A sharper test: take a sample with a clean image and heavily corrupted text, and measure $\\omega_{\\text{image}}$ and $\\omega_{\\text{text}}$ as corruption increases; the mechanism predicts a monotone shift toward the image, while weights that stay near uniform would show DMS contributes no dynamic behavior.","tokens_in":7508,"feed_emoji":"⚖️","tokens_out":7141,"duration_ms":74806,"temperature":0.7,"pith_summary":"Dynamic Modality Scheduling (DMS) is a per-sample reweighting rule: for each input stream, the paper computes a confidence score from predictive entropy, an epistemic-uncertainty score from Monte Carlo dropout, and a semantic-consistency score from cosine similarity to the other modalities, then sets softmax weights $\\omega_m \\propto \\exp(\\alpha c_m - \\beta u_m + \\gamma s_m)$ and forms $h=\\sum_m \\omega_m f^{(m)}$. The paper claims that inserting this scheduler at the encoder-output level improves vision-language tasks: VQA accuracy rises by 2.3 points, image-text retrieval Recall@1 rises by 3.1 points, and degradation under image or text noise is roughly halved relative to static fusion. If the claim holds, instance-level soft weighting is a lightweight way to make multimodal large models robust to noisy or missing modalities, and no retraining of the underlying backbones is required.","feed_headline":"Per-sample modality weights lift VQA accuracy 2.3%","feed_subtitle":"DMS rewards confident, aligned modalities and discounts noisy ones, cutting image-noise degradation from 21.4% to 11.5%.","key_machinery":"The load-bearing object is the weighted-sum fusion rule $h=\\sum_m \\omega_m f^{(m)}$ where $f^{(m)}$ is the encoder output for modality $m$ and $\\omega_m$ comes from a softmax over $\\alpha c_m - \\beta u_m + \\gamma s_m$. Here $c_m$ is $1-H(p^{(m)})$ with $H$ the entropy of the modality's softmax prediction, $u_m$ is the mean variance across $K$ classes and $T$ Monte Carlo dropout forward passes, and $s_m$ is the cosine similarity between $f^{(m)}$ and the average of the other modalities. The same weights feed a Modality Weight Consistency Loss $\\mathcal{L}_{\\text{mwcl}}=\\sum_m \\omega_m \\lVert h - f^{(m)}\\rVert_2^2$, which the paper derives as variance-reducing regularization that keeps the fusion near each unimodal embedding in proportion to its assigned weight. This rule is the entire mechanism: it converts three per-modality quality estimates into one normalized weight vector, and the paper's theoretical sections give a Lipschitz-based bound on deviation from oracle static fusion and a Rademacher-complexity generalization bound for the weighted hypothesis class.","core_discovery":"The paper's central discovery is that a soft, data-dependent weighting of pretrained unimodal embeddings, driven by three complementary signals, outperforms static blending and attention-based fusion on both clean and corrupted inputs. DMS posits that a modality which is confident (low predictive entropy), stable under stochastic dropout passes (low Monte Carlo dropout variance), and semantically aligned with the other modality should dominate the fused representation, while a noisy or contradictory modality should be downweighted. Experiments on VQA v2, MSCOCO captioning, and Flickr30K retrieval report DMS at 74.4 VQA accuracy versus 72.1 for static BLIP-2, and 61.5 versus 58.4 Recall@1; under Gaussian blur or text corruption, DMS retains 88.5% of clean VQA accuracy while the static baseline retains 78.6%. The authors present this as evidence that dynamic scheduling, rather than a change in fusion architecture, carries the robustness gain.","pith_inferences":["The paper leaves implicit that the confidence term $c_m$ presupposes a per-modality classification head with a softmax over $K$ classes, which does not exist naturally for generative tasks such as captioning; making DMS fully model-agnostic would require auxiliary heads or proxy confidence from the language model's token probabilities.","Because $\\mathcal{L}_{\\text{mwcl}}$ pulls the fused $h$ toward each unimodal embedding with weight $\\omega_m$, settings where one modality is almost completely dominant could over-regularize the fusion toward that modality and suppress cross-modal complementarity; this is testable by checking whether DMS underperforms static fusion when both modalities are clean but semantically redundant.","An economical extension would replace the $T$ Monte Carlo dropout passes with test-time augmentation or a lightweight variance proxy, cutting per-sample compute cost; the reported robustness gains suggest uncertainty estimates, not the specific estimator, are the important signal."],"forward_implications":["On standard benchmarks, DMS reports VQA accuracy 74.4 versus 72.1 for static BLIP-2, and Recall@1 61.5 versus 58.4, so instance-level weighting alone yields these gains.","Under image noise, DMS keeps 88.5% of clean VQA accuracy versus 78.6% for static fusion; under text noise degradation drops from -18.0% to -9.4%, implying the scheduler detects and discounts corrupted inputs.","Ablations attribute part of the gain to each factor: removing semantic alignment costs the most on retrieval (R@1 falls from 61.5 to 57.7), so cross-modal consistency carries a large part of the retrieval benefit.","The same scheduler transfers to LLaVA, raising factuality from 72.4 to 78.9 in open-ended question answering, supporting the paper's model-agnostic claim."],"supporting_citations":[{"why":"Supplies the BLIP-2 backbone and Q-Former fusion design that DMS claims to augment.","marker":"[1]"},{"why":"Supplies the LLaVA backbone used for the open-ended QA portability experiment.","marker":"[4]"},{"why":"Supplies Monte Carlo dropout as the uncertainty estimator used in the scheduler.","marker":"[15]"},{"why":"Supplies the CLIP image encoder that yields the visual embeddings DMS reweights.","marker":"[5]"},{"why":"Represents the static early-fusion or co-attention fusion approach DMS compares against.","marker":"[13]"},{"why":"Motivates the class of frozen-encoder multimodal large models DMS is designed to plug into.","marker":"[2]"}],"fun_headline_variants":["Dynamic modality weights lift VQA 2.3 points over static fusion","Second-guess noise: scheduler rewards confident modalities","Adaptive scheduling halves image-corruption loss in MLLMs","Confidence, uncertainty, and consistency drive modality scheduling","Per-sample modality weighting boosts robustness in multimodal LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an MLLM's fusion reduces to a weighted sum $h=\\sum_m \\omega_m f^{(m)}$ of unimodal encoder outputs, but BLIP-2 fuses through Q-Former cross-attention and LLaVA injects image tokens into the LLM, and the paper never shows how those architectures become such a weighted sum.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic modality weights lift VQA 2.3 points over static fusion","Second-guess noise: scheduler rewards confident modalities","Adaptive scheduling halves image-corruption loss in MLLMs","Confidence, uncertainty, and consistency drive modality scheduling","Per-sample modality weighting boosts robustness in multimodal LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1466,"prompt_tokens":1024,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":640,"tokens_out":442,"duration_ms":5765,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:43:29.907133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact pipeline on a standard BLIP-2 checkpoint without removing the Q-Former: if inserting DMS weights before the multimodal transformer does not reproduce roughly 74.4 VQA accuracy and the reduced noise degradation, the claim fails. A sharper test: take a sample with a clean image and heavily corrupted text, and measure $\\omega_{\\text{image}}$ and $\\omega_{\\text{text}}$ as corruption increases; the mechanism predicts a monotone shift toward the image, while weights that stay near uniform would show DMS contributes no dynamic behavior.","supporting_citations":[{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.ICML, 2016","cited_arxiv_id":null,"evidence_quote":"Supplies Monte Carlo dropout as the uncertainty estimator used in the scheduler."},{"cited_title":"Learning transferable visual models from natural language supervision.ICML,","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP image encoder that yields the visual embeddings DMS reweights."},{"cited_title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","cited_arxiv_id":null,"evidence_quote":"Represents the static early-fusion or co-attention fusion approach DMS compares against."}],"review_version":1}