{"id":"539dbd0d-637f-4040-9e2d-c7a92f445ab1","arxiv_id":"2506.17903","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CEDO, a combination of modality-specific learning rates, Pareto gradient synergy, and loss rescaling, reports improved accuracy on five Med-VQA benchmarks including two new biased splits.","lead":"A new training framework for medical visual question answering, called CEDO, adds three optimization tricks to cut language-bias shortcuts: separate learning rates for each input type, Pareto-style gradient balancing, and loss rescaling for rare answers. The authors report top accuracy on five benchmarks, including two newly constructed bias-sensitive datasets, and argue this makes medical VQA more reliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GMS gradient equations in §3.3 are internally inconsistent: Eq. (8) makes the image unimodal gradient identically zero and Eq. (12) is not an orthogonalization, so the reported gains cannot yet be attributed to the described debiasing mechanism.","rationale":"The reader identifies the same weakest assumption: the gradient definitions in Eq. (8) and the projection in Eq. (12) are inconsistent with the claimed mechanism. I agree, and I consider this the most load-bearing concern because the paper's contribution is not merely a number but a specific cause-effect debiasing mechanism. If the equations are taken literally, GMS cannot function as described; if they are typos, the actual algorithm is absent from the manuscript, making the reported gains unverifiable as stated. This is an internal consistency problem, not a disagreement with the field consensus. The concern is addressable: the authors can supply the exact gradient computation, correct the notation, and rerun the ablation. Given the absence of released code and the missing DeBCF comparison noted by the reader, conditional acceptance pending clarification remains the appropriate verdict. I therefore do not change the reader's decision, while underscoring that the GMS specification is the key obstacle to accepting the central claim as written.","tokens_in":12656,"tokens_out":5985,"duration_ms":68694,"concrete_test":"Obtain the exact gradient computation from the authors (pseudocode or released code) and rerun SLAKE-CP with GMS implemented two ways: (a) literal Eq. (8), i.e., gradients with respect to θ_q only, and (b) full-parameter gradients followed by true Gram-Schmidt orthogonalization, G'_q = G_q − ((G_q · G_v)/||G_v||²) G_v. If version (a) reproduces the reported 79.27, then GMS is inert and the improvement is misattributed; if only version (b) reproduces it, the printed equations must be corrected; if neither reproduces it, the reported result is not reproducible from the described method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical, but the mechanism is underspecified in a way that matters. Eq. (8) defines G_k = ∇_{θ_q} L_k(R_k, A) for all k ∈ {t, q, v}. The image unimodal loss L_v depends on θ_v and the shared fusion/classifier parameters, not θ_q, so G_v ≡ 0. Then Eq. (12), G'_q = G_q − proj_{G_q}(G_v), reduces to G'_q = G_q, and the gradient-orthogonality step does nothing. If the intended definition is G_k = ∇_θ L_k over the full parameter set, the paper never states it. Even with that correction, Eq. (11) is not a Gram-Schmidt removal of the biased component: the projection of G_v onto G_q is P = ((G_v · G_q)/||G_q||²) G_q, and subtracting it from G_q does not make G'_q orthogonal to G_v. In fact, if G_q = G_v, then G'_q = 0, erasing an agreed update rather than a biased one. Because the paper attributes the 2.85 and 2.62 point CP gains to this GMS mechanism, the current manuscript does not specify an algorithm that matches its own causal narrative. The ablation results in Table 4 provide some supporting evidence that each module helps, but without a faithful specification it is unclear what GMS actually computes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CEDO, a training-time optimization wrapper for Med-VQA that targets language bias from what it calls cause and effect sides. The method combines three modules: MHO, which assigns separate learning rates to question, image, and classifier parameters; GMS, which uses a Pareto-style combination of unimodal and joint gradients together with a gradient-projection correction; and DLR, which reweights the supervised contrastive loss according to inverse question-type/answer co-occurrence counts. The authors construct two bias-sensitive benchmarks, SLAKE-CP and VQA-RAD-CP, by redistributing samples per question type, and report state-of-the-art accuracy on SLAKE, VQA-RAD, their CP variants, and VQA-CE, with the largest gains on the CP benchmarks. The central claim is that this optimization wrapper comprehensively mitigates language biases while preserving or slightly improving standard accuracy.","tokens_in":13007,"tokens_out":4137,"duration_ms":47056,"significance":"If the proposed mechanism worked as described, CEDO would be a practical contribution: it is an optimization-level wrapper that does not require architecture changes, counterfactual sample generation, or additional inference branches, and it reports consistent gains on both in-distribution and out-of-distribution Med-VQA benchmarks. The ablation study in Table 4 gives initial evidence that each module contributes, and the experiments span five datasets including the natural-domain VQA-CE. However, the current manuscript does not provide a faithful specification of the GMS mechanism in Eqs. (8)-(13), and the headline differences are not supported by error bars or a validation protocol that avoids tuning on the test benchmark. The benchmark construction also needs clearer differentiation from the DeBCF protocol described in [Zhan et al., 2023]. For these reasons, the empirical claim is promising but cannot yet be fully credited as stated.","major_comments":[{"comment":"Eq. (8) defines G_k = ∇_{θ_q} L_k(R_k, A) for all k ∈ {t, q, v}, but the image unimodal loss L_v depends on θ_v and on the shared fusion and classifier parameters, not on θ_q; taken literally this makes G_v ≡ 0, so the Pareto combination in Eq. (9) does not balance modality gradients and the projection in Eq. (12) becomes degenerate. If the intended definition is a gradient over the full parameter set, that definition is never stated, and the projection in Eq. (11) is not a valid orthogonalization: subtracting proj_{G_q}(G_v) from G_q does not make G'_q orthogonal to G_v, and in the aligned case G_q = G_v it erases the entire agreed update rather than removing a biased component. Because the reported 2.85 and 2.62 point CP gains are attributed to this GMS mechanism, the paper currently does not specify a well-defined debiasing algorithm that matches its own causal narrative.","section":"§3.3, Eqs. (8)-(13)"},{"comment":"The reported improvements are not accompanied by error bars or significance tests, and the hyperparameters η_q, η_v, and η_c are selected on SLAKE-CP itself (Fig. 4), which is the same benchmark used for the headline comparison in Table 1. This makes it difficult to assess whether the 0.67-point gain on SLAKE or the 1.23-point gain on VQA-CE is distinguishable from run-to-run variation, and whether part of the SLAKE-CP advantage comes from test-set tuning. Please report multiple-seed means and standard deviations and move hyperparameter selection to a validation split.","section":"§4.3, Tables 1-3; §4.5, Fig. 4"},{"comment":"The section says the CP datasets are built \"following [Zhan et al., 2023]\" and at the same time calls them \"novel,\" so the relationship to the DeBCF bias-sensitive dataset should be clarified. In particular, the authors should state what is new relative to DeBCF, whether the same split protocol is reused, and how question types and \"Other\" answers are counted in the 39:1, 1:39, and 3:1 allocations. Without this information, the reader cannot judge whether the benchmarks are genuinely new or whether the evaluation is partly self-referential with respect to the dataset construction protocol.","section":"§4.2, Bias Reconstruction"}],"minor_comments":[{"comment":"The positive set is defined as P_i = {p ∈ I s.t. a_p = a_j}, but j is not defined in the formula; it should presumably be a_i, the ground-truth answer of sample i.","section":"§3.4, Eq. (16)"},{"comment":"The RUBi row on VQA-RAD-CP reports All=81.27 with Open=60.62 and Closed=86.13; please clarify whether the All column is a weighted average over the open and closed subsets or a separately computed metric, since the three values are not mutually consistent without additional information.","section":"Table 1, RUBi row"},{"comment":"The axis labels of Fig. 4 appear to show η_q × 1000, η_v × 1000, and η_c × 1000; the caption should state explicitly that the displayed values are the true learning rates divided by 1000, and the text should indicate which hyperparameters are held fixed while each one is varied.","section":"Fig. 4 and §4.5"},{"comment":"The paper does not state whether the constructed SLAKE-CP and VQA-RAD-CP splits and the training code will be released; for a benchmark contribution this information is important for reproducibility and for comparisons by later work.","section":"§4.2 and Datasets"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the empirical story is coherent, but the mechanism section doesn't survive a close read. Eq. (8) defines every unimodal gradient G_k as ∇_{θ_q} L_k, with k ranging over t, q, and v. That makes G_v identically zero, since the image-only loss does not depend on question parameters. Eq. (12) then subtracts the projection of a zero vector, so the “gradient orthogonality” step does nothing. If the authors meant the full parameter set, they never say so, and the formula still isn't an orthogonalization: subtracting the projection of G_v onto G_q from G_q leaves G'_q · G_v ≠ 0 in general, and when G_q = G_v you get G'_q = 0. This is not a cosmetic typo: GMS is the module the 2.85 and 2.62 point CP gains are attributed to, and the ablation in Table 4 credits GMS alone with 1.91 points.\n\nWhat's genuinely new is the specific combination of the three modules and the two bias-sensitive splits, SLAKE-CP and VQA-RAD-CP, that could be reused by the community. The reported results are consistent across standard, medical OOD, and natural OOD benchmarks, and the ablations show each component helps. The authors also cite the sources of each building block, which is honest about the borrowings.\n\nThe soft spots beyond the gradient issue: no code, no released data splits, no error bars or significance tests, and no direct comparison with DeBCF, the Med-VQA debiasing method the authors themselves identify as the first attempt. That omission matters because DeBCF is the main domain-specific baseline. Also, the three learning-rate hyperparameters are tuned on SLAKE-CP, so the CP gains are partly a selection result—common in this line, but worth stating.\n\nReader: anyone working on Med-VQA debiasing who wants a candidate wrapper and two new evaluative splits. But until the GMS equations are corrected and artifacts are released, I would not trust the stated mechanism. I would not desk-reject this: the fix is likely feasible, and the datasets alone have value. Send it to peer review with a clear requirement that the authors specify the actual gradient computation, fix the orthogonality step, release code and splits, and add DeBCF as a baseline.","headline":"The empirical wrapper looks useful, but the GMS gradient equations are internally inconsistent, so the debiasing mechanism as written cannot explain the gains.","tokens_in":13508,"tokens_out":3142,"would_cite":false,"duration_ms":34439,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training-time optimization framework, CEDO, can remove language biases in medical visual question answering by jointly correcting answer-category imbalance and question-modality shortcut, and reports…","keywords":["medical visual question answering","language bias","bias mitigation","multimodal learning","gradient orthogonality","Pareto optimization","loss rescaling","out-of-distribution generalization"],"falsifier":"On a batch of SLAKE-CP, compute the cosine similarity between $G_q$ and $G_v$ and the norm of $G_q$ before and after the orthogonality projection. If, averaged over batches, the cosine similarity is already near zero—or the projection barely changes $G_q$—then the orthogonality step has nothing to remove, and any accuracy gain must come from MHO or DLR rather than GMS.","tokens_in":12457,"feed_emoji":"🩺","tokens_out":8220,"duration_ms":78256,"temperature":0.7,"pith_summary":"The paper claims that language biases in medical visual question answering (Med-VQA) can be mitigated by a training-time optimization framework, CEDO, that treats the bias's cause (answer-category imbalance) and its effect (the question modality becoming a shortcut) separately. On two newly constructed out-of-distribution benchmarks, SLAKE-CP and VQA-RAD-CP, CEDO improves accuracy by 2.85 and 2.62 points over the state of the art, while also raising accuracy on the standard SLAKE and VQA-RAD datasets. The appeal is that a generic wrapper around an existing Med-VQA model can make it more robust to spurious question-answer correlations without losing in-distribution performance.","feed_headline":"CEDO cuts medical VQA language bias, gains 2.85 points","feed_subtitle":"A pure training-time fix lifts accuracy on biased and standard medical VQA benchmarks.","key_machinery":"The load-bearing mechanism is a three-part gradient and loss transformation. First, MHO partitions the model parameters into question, image, and classifier groups and updates each with its own learning rate, slowing the bias-prone question encoder. Second, GMS computes joint and unimodal losses, combines their gradients with adaptive Pareto weights, then replaces each gradient by itself minus its projection onto the others, an orthogonality constraint intended to delete the bias component from the question gradient. Third, DLR multiplies the supervised contrastive term by a weight $W_j^i = \\log(1+\\exp(1/(M_j m_j^i)))$ per sample, where $M_j$ is the number of samples of question type $j$ and $m_j^i$ the number of answers of category $i$ under that type, so rare answers are not starved. Together these modules are meant to prevent the question encoder from dominating and keep the loss balanced across answer categories.","core_discovery":"CEDO is built on the observation that a Med-VQA model's answers are distorted by two linked biases: imbalanced answer distributions cause the question type to correlate spuriously with frequent answers, and during training the question encoder receives oversized gradient updates that turn it into a shortcut. The framework attacks both sides: MHO assigns a lower learning rate to the question encoder and higher rates to the image encoder and classifier; GMS combines the joint and unimodal losses with a Pareto-optimized weighted gradient and then subtracts the projection of each gradient onto the others to enforce orthogonality; DLR reweights the supervised contrastive loss by an inverse question-type/answer frequency with a softplus smoothing. After integrating these into the base model, the paper reports state-of-the-art results on biased benchmarks (79.27 on SLAKE-CP, 92.07 on VQA-RAD-CP) and gains of 1.98 and 0.67 points on the standard SLAKE and VQA-RAD datasets.","pith_inferences":["The cause/effect decomposition suggests a general recipe for debiasing other multimodal systems: correct the data-level imbalance before it enters the loss, and constrain gradient updates to stop the dominant modality from absorbing the bias.","If the gradient-orthogonality step is really doing the work, the same projection could be applied to other bias-carrying gradient directions (e.g., location priors in chest X-ray diagnosis) as long as the bias direction can be identified.","One way to test the mechanism directly is to ablate the projection direction: replace it with a random unit vector of the same norm; if accuracy does not drop, the orthogonality constraint is not the active ingredient.","The reported parameter analysis keeps $\\eta_q$ smaller than $\\eta_v$ and $\\eta_c$ (0.002 vs 0.003), an asymmetry consistent with the paper's story and a cheap sensitivity check: sweep the three rates independently and see if the advantage disappears when $\\eta_q \\ge \\eta_v$."],"forward_implications":["If CEDO is correct, a base Med-VQA model can be made more robust to language bias purely by changing its optimization, without altering its architecture or training data.","The new benchmarks SLAKE-CP and VQA-RAD-CP provide a repeatable protocol for measuring language-bias robustness in the medical domain, following the VQA-CP v2 ratio.","The three modules are additive: ablations show each alone improves SLAKE-CP (MHO +1.59, GMS +1.91, DLR +2.52) and the full combination +2.85 over baseline, suggesting the cause/effect split captures complementary failure modes.","The paper reports that CEDO also improves accuracy on the large-scale natural-domain VQA-CE benchmark, indicating the method is not limited to the two constructed datasets."],"supporting_citations":[{"why":"The base model whose total loss and architecture CEDO is integrated into; all comparisons and ablations are on top of it.","marker":"[Basu et al., 2023]"},{"why":"Supplies the Pareto method for combining competing gradients in the GMS module.","marker":"[Sener and Koltun, 2018]"},{"why":"Provides the theoretical properties (Pareto-stationarity and descent direction) that justify the Pareto integration step.","marker":"[Désidéri, 2012]"},{"why":"Inspiration for the distribution-adapted loss rescaling used in DLR.","marker":"[Guo et al., 2021]"},{"why":"The counterfactual-training approach that CEDO compares against, and the basis for constructing the SLAKE-CP and VQA-RAD-CP benchmarks.","marker":"[Zhan et al., 2023]"},{"why":"Defines the supervised contrastive loss that DLR reweights.","marker":"[Khosla et al., 2020]"},{"why":"Establishes the VQA-CP v2 splitting ratios (39:1, 1:39, 3:1) used to build the biased benchmarks.","marker":"[Agrawal et al., 2018]"}],"fun_headline_variants":["Training-time CEDO: less bias, more accuracy in Med-VQA","CEDO tames language bias in medical VQA, boosts scores","No more shortcuts: CEDO optimizes cause-effect for robust Med-VQA","Cause-effect tweak CEDO cuts Med-VQA bias, lifts benchmarks","Adaptive training CEDO reduces medical VQA bias, raises accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole mechanism hinges on the gradients used in the Pareto combination actually being defined for each modality's own parameters, and on the projection step removing only the bias component of the question gradient rather than useful signal.","fun_headline_variants_meta":{"raw":{"variants":["Training-time CEDO: less bias, more accuracy in Med-VQA","CEDO tames language bias in medical VQA, boosts scores","No more shortcuts: CEDO optimizes cause-effect for robust Med-VQA","Cause-effect tweak CEDO cuts Med-VQA bias, lifts benchmarks","Adaptive training CEDO reduces medical VQA bias, raises accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2931,"prompt_tokens":951,"completion_tokens":1980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1881}},"tokens_in":567,"tokens_out":1980,"duration_ms":16470,"temperature":1.0,"reasoning_tokens":1881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:44.016029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a batch of SLAKE-CP, compute the cosine similarity between $G_q$ and $G_v$ and the norm of $G_q$ before and after the orthogonality projection. If, averaged over batches, the cosine similarity is already near zero—or the projection barely changes $G_q$—then the orthogonality step has nothing to remove, and any accuracy gain must come from MHO or DLR rather than GMS.","supporting_citations":[{"cited_title":"Venkatesh Babu","cited_arxiv_id":null,"evidence_quote":"The base model whose total loss and architecture CEDO is integrated into; all comparisons and ablations are on top of it."},{"cited_title":"Multi-task learning as multi-objective optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the Pareto method for combining competing gradients in the GMS module."},{"cited_title":"Loss re-scaling vqa: Re- visiting the language prior problem from a class-imbalance view.IEEE Transactions on Image Processing (TIP), 31:227–238,","cited_arxiv_id":null,"evidence_quote":"Inspiration for the distribution-adapted loss rescaling used in DLR."},{"cited_title":"Debias- ing medical visual question answering via counterfactual training","cited_arxiv_id":null,"evidence_quote":"The counterfactual-training approach that CEDO compares against, and the basis for constructing the SLAKE-CP and VQA-RAD-CP benchmarks."},{"cited_title":"Supervised contrastive learning","cited_arxiv_id":null,"evidence_quote":"Defines the supervised contrastive loss that DLR reweights."},{"cited_title":"Don’t just assume; look and answer: Overcoming priors for visual question answering","cited_arxiv_id":null,"evidence_quote":"Establishes the VQA-CP v2 splitting ratios (39:1, 1:39, 3:1) used to build the biased benchmarks."}],"review_version":1}