{"id":"ac72854b-06e5-455f-a085-56f2ba04aac8","arxiv_id":"2501.15955","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Pre-training label imbalance persists in foundation models after PEFT fine-tuning, cannot be fixed by logit adjustment, and averaging LA-adjusted logits from three CLIP variants improves long-tailed accuracy.","lead":"This paper studies what happens when a vision foundation model trained on imbalanced data is fine-tuned on another long-tailed dataset, and claims the bias baked into the pretrained weights matters more than the imbalance in your own data. It proposes a 'backdoor adjustment' that treats three different CLIP models as incomplete views of the same images and averages their logit-adjusted predictions, reporting about 1.7% average accuracy gain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The backdoor adjustment is an equal-weight ensemble over three hand-picked checkpoints; no identification of C or test of P(c)=1/M supports the causal reading, so the central claim that the method 'learns the true causal effect' is unsubstantiated.","rationale":"The reader's conditional verdict is appropriate and my stress pass does not move it. The paper's empirical observations — that different CLIP variants have different class-conditional strengths, that LA helps data imbalance but not the residual bias, and that ensembling three variants improves tail accuracy — are plausible and partially supported by the included ablations and synthetic control (Secs. E.6, E.7, E.9). Those are real evidence, and the code release is a plus. However, the central novel contribution is the causal backdoor adjustment, and that contribution is exactly where the paper is weakest. Eq. (7) is algebraically an average over three models; calling it P(Y|do(x)) requires an identified confounder C and a correct prior P(c). Neither is supplied, and the paper's own definition of C is informal ('incomplete semantic factor'). The three chosen models are not sampled from any defined distribution, and M=3 is a modeling choice, not a sample from P(C). A validation-weight check would directly test the uniform prior; a same-backbone control would test whether cross-model diversity is necessary. Both are inexpensive and would substantially raise confidence if they pass. Without them, the causal language outruns the evidence, so a conditional verdict — accept the empirical contribution, require identification evidence before accepting the causal claim — is the right call. I agree with the reader that this is the weakest assumption, and I see no additional internal inconsistency that would warrant a harder rejection.","tokens_in":20467,"tokens_out":7939,"duration_ms":86196,"concrete_test":"Run a weight-sensitivity check on the released code: on a held-out validation split of ImageNet-LT (and Places365-LT), fit convex fusion weights w_c over the three LA-adjusted logits by maximizing balanced accuracy (simplex grid search or a learned softmax head), and compare with the fixed uniform weights 1/3 used in Eq. (7). If the optimal weights deviate substantially from 1/3 and improve accuracy beyond run-to-run noise, then P(c)=1/M is rejected and the backdoor estimate is misspecified. If the optimal weights are nearly uniform, the equal-weight choice is empirically benign, but the causal claim would still require a separate identification argument, e.g., a known-P(C) synthetic experiment in the style of Sec. E.9.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. (7) in Sec. 4.2: P(Y|do(x)) = Σ_c P(Y|b,c)P(c), with P(c)=1/M and the three values of C approximated by CLIP, OpenCLIP, and MetaCLIP. For this to be a backdoor adjustment rather than an equal-weight ensemble, C must be a well-defined confounder with a known distribution, and these three checkpoints must be draws (or at least representative supports) of that distribution. The paper provides no constructive definition of C, no measurement model linking a model identity to a value of C, and no argument that three public checkpoints are exchangeable or that uniform weights are correct. The M-ablation in Tab. 8 only shows that adding more models helps, which is generic ensembling behavior and is equally consistent with C being irrelevant. The Grad-CAM evidence (Fig. 4, Sec. E.6) shows that the models attend differently, but differential attention is not identification of a confounder; it is diversity. Consequently, the central claim that the method 'learns the true causal effect' is not established, and the reported 1.67% average gain may be entirely due to ensembling three LA-debiased classifiers. The empirical decomposition of parameter vs data imbalance can still be true; what fails is the causal interpretation of the proposed estimator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how pretraining imbalance in foundation models affects long-tailed downstream classification. It distinguishes parameter imbalance (bias inherited from imbalanced pretraining data, estimated through a GLA-based prior) from data imbalance (bias from the downstream long-tailed training set), and argues empirically that parameter imbalance is more influential and cannot be fixed by logit adjustment. The proposed method, framed as a backdoor adjustment, treats an 'incomplete semantic factor' C as a confounder and averages LA-adjusted predictions from CLIP, OpenCLIP, and MetaCLIP. Experiments on ImageNet-LT, Places365-LT, and iNaturalist2018 show consistent accuracy gains over PEFT baselines, with additional ablations over M and PEFT variants.","tokens_in":20783,"tokens_out":6684,"duration_ms":64845,"significance":"The paper contains a useful and reproducible empirical study: the finding that logit-adjustment improves the classifier more than the representation (Tab. 4), and the observation that GLA-Train does not remove the pretraining bias (Tab. 3), are informative for practitioners. The proposed method is simple, code is released, and the reported gains are consistent across three large long-tailed benchmarks and two PEFT methods. However, the central advertised contribution is the causal claim that the method 'learns the true causal effect between input samples and labels.' That claim is not supported: the estimator in Eq. (7) reduces to an equal-weight ensemble of LA-adjusted models, and no identification argument for C or the uniform prior is provided. The empirical method may still be valuable as a diversified ensemble, but the causal interpretation is the load-bearing part of the paper's framing and is currently unsubstantiated.","major_comments":[{"comment":"The backdoor adjustment in Eq. (7) requires C to be a well-defined confounder with a known distribution, but no constructive definition or measurement model for C is given. The three foundation models are treated as instantiations of C, yet model identity is not a value of a semantic factor; the Grad-CAM evidence in Fig. 4 and Sec. E.6 shows differential attention, which demonstrates diversity, not identification. Consequently, with P(c)=1/M, Eq. (7) is an equal-weight average of LA-adjusted logits, and the Abstract's claim that the method 'learns the true causal effect' is not established. A necessary control is a baseline that applies the same LA adjustment to each of the three models and simply averages their logits, without the causal framing.","section":"Sec. 4.2, Eq. (7)"},{"comment":"The uniform prior P(c)=1/M and the exchangeability of CLIP, OpenCLIP, and MetaCLIP are load-bearing assumptions, but they are not tested or defended. The M-ablation in Tab. 8 only shows that adding more foundation models improves accuracy, which is also the generic behavior of an ensemble and is compatible with a model in which C has no causal role. Without a check of the prior or an identification argument for C, the method cannot be described as a backdoor adjustment; it is an ensemble with equal weights.","section":"Sec. 4.2 and Tab. 8"},{"comment":"The empirical decomposition into parameter imbalance and data imbalance relies entirely on the GLA-estimated prior bPP(Y), both for the P-Many/P-Medium/P-Few groupings and for Definition 3.1. Since bPP(Y) is fitted on the validation set (Eq. 3) and the true PP(Y) is inaccessible, the paper should provide a sensitivity analysis showing that the qualitative conclusions in Fig. 3 and Tab. 2 are robust to errors in bPP(Y); otherwise the central claim that parameter imbalance 'plays a more critical role' could be an artifact of the estimator rather than a property of pretraining data.","section":"Sec. 3.1, Def. 3.1, Eq. (3)"},{"comment":"The synthetic experiment intended to verify that C is a confounder defines C explicitly as an object part in generated images, so it cannot validate the real-world identification of C across CLIP, OpenCLIP, and MetaCLIP. Moreover, the statement that a model trained on Atrain estimates P(Y|do(X)) is asserted rather than derived; balanced training over C is not an intervention in the causal sense, so the experiment does not supply the missing identification argument.","section":"Appendix E.9"}],"minor_comments":[{"comment":"The claimed 'average performance increase of about 1.67%' is not precisely defined; please specify which baselines and datasets are averaged, and report variance or significance if available.","section":"Abstract and Sec. 5"},{"comment":"The line 'It contains a total of 12.21K images' for ImageNet-LT is a factual error; ImageNet-LT has approximately 115.8K training images.","section":"Appendix A.1"},{"comment":"The Decoder row reports '21.26 34' under Params and Epochs; this appears to be a formatting error with missing units (presumably 21.26M and 34).","section":"Tab. 5"},{"comment":"The sentence 'As shown in Fig. 8, the result verifies our point' appears to refer to Fig. 9, which is the figure showing the comparison between training from scratch and fine-tuning.","section":"Appendix E.7"},{"comment":"There are several typographical errors in table captions and text, including 'Ther performance' in Tabs. 3 and 11 and 'Resverse' in Tab. 13; these should be corrected.","section":"Tabs. 3, 11, 13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the experiments are extensive, but the causal interpretation is the weakest part of the paper. I would want the revision to either supply real identification evidence for C and the uniform prior, or to explicitly reframe the method as a diversified ensemble of LA-adjusted models. The latter is an honest and still useful contribution, but it would require changing the title, abstract, and the way the method is presented throughout the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a solid empirical finding and a method that works modestly well, but the causal packaging doesn't survive contact with the equations. The parameter-imbalance vs data-imbalance distinction is genuinely useful, and the negative result that GLA-Train fails to fix parameter imbalance while LA fixes data imbalance is a real contribution. The proposed method—fine-tune CLIP, OpenCLIP, and MetaCLIP with logit adjustment and average their predictions—is simple and consistently beats single-model LA by roughly 1.5-3% on the three benchmarks. That part is reproducible and worth knowing.\n\nThe soft spot is the claim that this is a backdoor adjustment that 'learns the true causal effect.' Eq. (7) reduces to a uniform average over three LA-adjusted models. For it to be backdoor adjustment, C has to be a well-defined confounder with P(c)=1/M justified, and CLIP/OpenCLIP/MetaCLIP have to be instantiations or draws from that distribution. None of that is established. The paper defines C via the models, then uses the models to estimate P(Y|b,c), which is circular. The Grad-CAM and synthetic experiments in the appendix show that different models attend to different parts and that a known confounder can hurt, but they don't show these three checkpoints are exchangeable draws from a common C. So the causal interpretation is unsupported. The practical method survives—it's an ensemble of three LA-debiased classifiers, and the M-ablation shows adding models helps, which is exactly what you'd expect from ensembling. But the paper overclaims by calling it causal.\n\nTwo more issues: the estimated pretraining prior from GLA is load-bearing for the parameter-imbalance analysis and is fitted on the validation set without verification, and the main tables have no error bars. These are addressable. The missing baseline—plain average of the three LA models' logits without any causal framing—is the one that would let a reader decide whether the causal story adds anything beyond ensembling. Citation pattern is fine; they build directly on GLA and LA.\n\nWho this is for: practitioners fine-tuning CLIP-like models on long-tailed data will get a useful takeaway and a cheap method. The paper deserves a serious referee, but a referee should push hard on the causal claim and the missing ensemble baseline. If it's resubmitted, I'd want the causal language cut back to 'we ensemble diverse LA-fine-tuned models' unless identification is provided.","headline":"Useful empirical decomposition of upstream vs downstream imbalance in fine-tuned foundation models, but the 'backdoor adjustment' is really an equal-weight ensemble and the causal story should be cut or substantially rewritten.","tokens_in":21296,"tokens_out":2545,"would_cite":true,"duration_ms":23285,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Foundation models carry hidden bias from pre-training data that ordinary re-balancing cannot fix; the paper proposes a causal backdoor adjustment that averages three models and lifts accuracy ~1.67% per dataset.","keywords":["long-tailed learning","foundation models","parameter imbalance","data imbalance","causal inference","backdoor adjustment","logit adjustment","parameter-efficient fine-tuning"],"falsifier":"A simple test construction is given by the paper's own E.9 setup: train the same model on data generated with unbalanced incomplete semantic factors (e.g. 90 percent dog-head images) and observe a large performance drop on a balanced test set, which would confirm the confounder effect; conversely, if varying the prior of the incomplete semantic factors across models produces no drop and averaging three arbitrary models gives the same gain, the causal mechanism is not needed.","tokens_in":20240,"feed_emoji":"⚖️","tokens_out":3706,"duration_ms":28210,"temperature":0.7,"pith_summary":"The paper argues that when a foundation model is fine-tuned for a long-tailed downstream task, the model carries two distinct kinds of bias: data imbalance from the downstream training set and parameter imbalance inherited from the imbalanced pre-training data. The paper's central claim is that parameter imbalance dominates during fine-tuning and cannot be removed by standard re-balancing tricks such as logit adjustment, which only fix the classifier. To address both imbalances, the paper treats the incomplete semantic factor — the partial visual features a biased pre-trained model learns, like recognizing a dog only by its head — as a confounder, and applies a backdoor adjustment that averages predictions across multiple foundation models. The method reports consistent gains, roughly 1.67% average per dataset, with the largest improvements on tail classes.","feed_headline":"Averaging three CLIPs fixes a bias that re-balancing cannot","feed_subtitle":"They isolate pre-training imbalance as the real culprit and fix it with backdoor adjustment.","key_machinery":"The key machinery is backdoor adjustment from causal inference. The paper models an incomplete semantic factor $C$ — a partial feature captured by a biased pre-trained model, such as focusing on a dog's head instead of its whole body — as a confounder that opens the spurious path $X\\leftarrow C\\rightarrow Y$. By estimating $P(Y|\\operatorname{do}(x)) = \\sum_c P(Y|b,c)P(c)$ with balanced representation $b$, the method removes the confounding effect; with $P(c)=1/M$ and $M=3$ models, the operation is an equal-weight fusion of three fine-tuned, logit-adjusted models.","core_discovery":"The paper finds that fine-tuning a foundation model on a long-tailed task inherits two separate biases: data imbalance from the downstream data and parameter imbalance from the pre-training data. Parameter imbalance, measured by the estimated label prior of the inaccessible pre-training data, persists after fine-tuning and is not mitigated by logit adjustment or other re-balancing losses applied during training. The proposed remedy is a backdoor adjustment that estimates $P(Y|\\operatorname{do}(x))$ instead of $P(Y|x)$, treating the incomplete semantic factor $C$ as a confounder on the path $X\\leftarrow C\\rightarrow Y$. Concretely, the method fine-tunes three CLIP-family models (CLIP, OpenCLIP, MetaCLIP) with a re-balanced loss, treats each as an instantiation of an incomplete semantic factor, and averages their predictions under a uniform prior $P(c)=1/M$. The paper reports that this adjustment yields an average gain of about 1.67% per dataset over the strongest baselines on ImageNet-LT, Places365-LT, and iNaturalist2018.","pith_inferences":["Editorial inference: the uniform prior $P(c)=1/M$ makes the method an equal-weight ensemble; if the three models are not exchangeable draws from the confounder distribution, the causal language is not strictly needed to describe the result — averaging three logit-adjusted models may still work as an ensembling trick.","Editorial inference: the method could be tested against a cheaper alternative, e.g. fine-tuning one model with three different re-balancing seeds or three different PEFT adapters, to see whether the gains come from model diversity or from the causal adjustment itself.","Editorial inference: the paper's own E.4 experiments suggest that adding more tail data can mitigate parameter imbalance in the Reverse setting, implying a testable conjecture that data augmentation for tail classes could substitute for extra foundation models.","Editorial inference: because the method averages three fine-tuned models, its FLOPs in the paper grow linearly with $M$; the evaluation of trade-off between compute and accuracy gains is left open."],"forward_implications":["If parameter imbalance is the dominant bias in fine-tuned foundation models, then improving tail-class performance on a downstream task requires acting on pre-trained weights, not just on the downstream loss or classifier.","Logit adjustment and similar re-balancing techniques should be re-evaluated for PEFT-based long-tailed learning: they help data imbalance by improving the classifier but leave parameter imbalance largely untouched.","Ensembling multiple foundation models that have different pre-training data biases becomes a practical way to deconfound downstream predictions, with gains that grow as more diverse models are added.","The backdoor adjustment gives a concrete way to improve fairness across head and tail classes simultaneously, without the trade-off usually seen when re-balancing only the classifier.","The causal framing suggests that any future foundation model with known different pre-training imbalance could be plugged into this adjustment without retraining."],"supporting_citations":[{"why":"GLA provides the estimated pre-training label prior used to measure parameter imbalance and as a comparison baseline for adjustment techniques.","marker":"Zhu et al., 2024"},{"why":"LA supplies the logit adjustment method that the paper tests for data imbalance and that is used as the re-balancing component in the backdoor adjustment.","marker":"Menon et al., 2020"},{"why":"The backdoor criterion is the causal machinery that the paper invokes to justify estimating $P(Y|\\operatorname{do}(x))$.","marker":"Pearl, 2009"},{"why":"AdaptFormer is the PEFT method used to fine-tune the foundation models in the main experiments.","marker":"Chen et al., 2022"},{"why":"LIFT is the state-of-the-art PEFT-based long-tailed baseline that the paper improves upon and whose heavy-fine-tuning insight motivates the PEFT setting.","marker":"Shi et al., 2024"},{"why":"CLIP is one of the three foundation models whose pre-training data imbalance constitutes parameter imbalance.","marker":"Radford et al., 2021"},{"why":"OpenCLIP is the second foundation model, trained on LAION, used as another instance of incomplete semantic factor.","marker":"Cherti et al., 2023"},{"why":"MetaCLIP is the third foundation model, trained on MetaData, completing the $M=3$ approximation of incomplete semantic factors.","marker":"Xu et al., 2023a"}],"fun_headline_variants":["Three CLIPs beat re-balancing for long-tail bias","Why re-balancing fails: pre-training leaves a hidden bias","Backdoor adjustment fixes the bias re-balancing misses","Parameter bias, not data bias, survives fine-tuning","Averaging CLIP models cancels the confounder in long-tail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire causal interpretation rests on treating CLIP, OpenCLIP, and MetaCLIP as three draws from the confounder distribution $C$ with equal prior; if they are not exchangeable instantiations of the same incomplete semantic factor, the backdoor adjustment is just an equal-weight ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Three CLIPs beat re-balancing for long-tail bias","Why re-balancing fails: pre-training leaves a hidden bias","Backdoor adjustment fixes the bias re-balancing misses","Parameter bias, not data bias, survives fine-tuning","Averaging CLIP models cancels the confounder in long-tail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3301,"prompt_tokens":1007,"completion_tokens":2294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2220}},"tokens_in":623,"tokens_out":2294,"duration_ms":15317,"temperature":1.0,"reasoning_tokens":2220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:50:04.005634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A simple test construction is given by the paper's own E.9 setup: train the same model on data generated with unbalanced incomplete semantic factors (e.g. 90 percent dog-head images) and observe a large performance drop on a balanced test set, which would confirm the confounder effect; conversely, if varying the prior of the incomplete semantic factors across models produces no drop and averaging three arbitrary models gives the same gain, the causal mechanism is not needed.","supporting_citations":[{"cited_title":"Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models","cited_arxiv_id":null,"evidence_quote":"GLA provides the estimated pre-training label prior used to measure parameter imbalance and as a comparison baseline for adjustment techniques."},{"cited_title":"Long-tail learning with foundation model: Heavy fine-tuning hurts","cited_arxiv_id":null,"evidence_quote":"LIFT is the state-of-the-art PEFT-based long-tailed baseline that the paper improves upon and whose heavy-fine-tuning insight motivates the PEFT setting."}],"review_version":1}