{"id":"6cdbbaf2-51e3-4815-aceb-27be61f11b99","arxiv_id":"2411.17454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"FLEX-CLIP combines VAE-GAN feature generation with a gated residual fusion of CLIP features to improve few-shot cross-modal retrieval.","lead":"The paper proposes FLEX-CLIP, a method that generates pseudo image and text features with a VAE-GAN and then fuses them with original CLIP features using a learned gate, to improve zero- and few-shot cross-modal retrieval. The method is tested on four datasets and reported to outperform CLIP and prior methods, though no code or error bars are provided.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-1 training may leak target-domain samples: Section IV-C says the VAE-GAN is trained with source data and 'a small amount of target domain data,' contradicting the zero-shot setting in Section III-A; if real target features are used, the reported up-to-7.9% margin over CLIP is not a zero-shot…","rationale":"The reader's weakest_assumption (unseen generative generalization) is real, but my concern is more specific and more immediate: Section IV-C contains an explicit statement that the generation stage is trained with 'a small amount of target domain data,' which contradicts the zero-shot protocol defined in Section III-A and Section IV-A. This makes the strongest empirical claim—that FLEX-CLIP outperforms CLIP in zero-shot retrieval by up to 7.9 MAP—unverifiable from the text alone. If the statement is literal, the zero-shot results are invalid because target information was used during training. If it is imprecise and only class embeddings were used, the claim rests on the generative generalization assumption, which the paper does not test directly. In either case, the current evidence is insufficient to support the central claim. I therefore recommend UNVERDICTED rather than CONDITIONAL, because the issue is not a missing hyperparameter or error bar but a potential protocol violation that changes the meaning of the reported results. The concrete test I propose—rerunning zero-shot with a strict source-only Stage 1 and measuring the MAP gap—would settle whether the 7.9% victory over CLIP survives a clean zero-shot protocol.","tokens_in":36456,"tokens_out":4438,"duration_ms":50111,"concrete_test":"Obtain the exact data split and training configuration from the authors, or reimplement the protocol from Section IV. Run the zero-shot setting under two Stage-1 conditions: (A) strictly source-domain image/text features with only target class embeddings available, as required by Section III-A; (B) the protocol as written in Section IV-C that may use 'a small amount of target domain data.' Compare the resulting zero-shot MAP on Table III datasets. If condition A yields a drop larger than about 2 MAP on NUSWIDE or NUSWIDE-10K, or if FLEX-CLIP no longer beats CLIP, the reported 7.9% margin is attributable to target leakage.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's headline zero-shot claim (Table III: FLEX-CLIP beats CLIP by up to 7.9 MAP) depends on the generative model producing target-domain pseudo-features without seeing target-domain samples. Section IV-C states: 'In the first phase, the multimodal feature generation model is first trained with source domain data and a small amount of target domain data.' This directly conflicts with Section III-A, which defines zero-shot training as using only the source domain Xs, and with Section IV-D1, where the zero-shot results are reported. If that 'small amount of target domain data' includes real target-domain image or text features, then the zero-shot experiment is not zero-shot: the generator or projection network has already seen target information, and the comparison to the CLIP baseline is unfair. If the phrase instead refers only to target class embeddings, then the paper relies on the weaker assumption that a generator trained on source features and source class attributes can synthesize target features that match the real target distribution. The paper provides no distributional evidence for this, such as feature-space distance metrics or visualizations of generated versus real target features. Either way, the central zero-shot result is not currently supported: it is either an artifact of target leakage or an untested generalization assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLEX-CLIP, a two-stage method for X-shot (zero- and few-shot) cross-modal retrieval. Stage 1 trains a composite VAE-GAN to generate pseudo image/text features from class embeddings and latent codes, aiming to alleviate data imbalance. Stage 2 trains projection networks plus a gate residual network that adaptively fuses original CLIP features with projected features, aimed at mitigating feature degradation. Experiments on Wikipedia, Pascal Sentence, NUS-WIDE, and NUS-WIDE-10K are reported with MAP in 0-, 1-, 3-, and 5-shot settings, claiming improvements over CLIP and other baselines. The authors do not provide code, data splits, random seeds, or variance estimates.","tokens_in":36797,"tokens_out":2522,"duration_ms":24505,"significance":"If the reported improvements are reproducible and the zero-shot protocol is sound, the idea of feature-level VAE-GAN generation combined with gated fusion of CLIP features is a plausible contribution to few-shot cross-modal retrieval. The paper's explicit goal of improving upon raw CLIP features is interesting, and the gated residual fusion is a simple, potentially useful mechanism. However, the central zero-shot claim is currently compromised by a protocol ambiguity and by the lack of distributional evidence for the generated features, and the experimental reporting is insufficient for assessing robustness. With the requested clarifications and additional analysis, the method could be a meaningful advance; as presented, the evidence is not yet convincing.","major_comments":[{"comment":"The zero-shot definition in Section III-A states that in the zero-shot scenario the model 'can only be trained by using the samples in the source domain Xs.' However, Section IV-C states that in the first phase, the multimodal feature generation model is 'trained with source domain data and a small amount of target domain data.' If that small amount includes real target-domain image/text features, the zero-shot results in Table III are not zero-shot because target information has been seen during training, making the comparison with the CLIP baseline unfair. If it refers only to target class embeddings (attributes), this must be stated explicitly, because the current wording is in direct conflict with the problem formulation.","section":"Section IV-C vs. Section III-A"},{"comment":"The load-bearing premise that a VAE-GAN trained on source features and class embeddings can synthesize target-domain pseudo-features whose distribution is close enough to real target features to improve the downstream projection is not verified. The paper provides no quantitative evidence such as feature-space distance (e.g., FID or MMD) between generated and real target features, no visualization of generated versus real features, and no ablation that replaces pseudo-features with real target features (or varies the number of real target features used in stage 1). Without such evidence, the zero-shot improvements in Table III are not supported, and it remains unclear whether the pseudo-features are the cause of the gains.","section":"Section III-D and Tables III-IV"},{"comment":"The abstract claims 'a 7%-15% improvement over state-of-the-art methods,' while the contributions section claims 'by up to 7.9%.' The 7.9% figure appears to be the best improvement over the CLIP baseline on a single dataset (Table III, NUS-WIDE), not over the best state-of-the-art method, and the 7%-15% range is not substantiated by any table. The paper should reconcile these numbers and state clearly which comparison (which baseline, which dataset, which setting) supports each claim.","section":"Abstract and Contributions vs. Results"},{"comment":"The experimental protocol is under-specified and the reported improvements may be explained by an unfair comparison. The CLIP baseline is not fine-tuned in the reported experiments ('the 1,3,5-shot experiment results are same in CLIP'), whereas FLEX-CLIP trains projection networks and a gate network on the training set, meaning the comparison is between a frozen zero-shot CLIP and a model that has been trained on source (and possibly target) data. To make the comparison meaningful, the paper should either fine-tune the CLIP baseline under the same X-shot protocol or restrict claims of 'surpassing CLIP' to the specific setting where CLIP is not trained, and also report a CLIP baseline that uses the same trained projectors without the generation or gating components.","section":"Section IV-D and Implementation Details"},{"comment":"The zero-shot results in Table III show FLEX-CLIP outperforming CLIP by substantial margins on NUS-WIDE-10K and NUS-WIDE (e.g., 0.579 vs. 0.503 average MAP on NUS-WIDE-10K). Given the protocol ambiguity in Section IV-C, it is critical to report the exact number of target-domain samples used in the first stage for each dataset and setting, and to release the data splits and random seeds so that the zero-shot condition can be verified. If the current numbers were obtained with target-domain real samples, the zero-shot claims should be retracted or re-run under the strict protocol of Section III-A.","section":"Tables III and IV, and Section IV-D1"}],"minor_comments":[{"comment":"The text references 'Table 1' and 'Table 2' when discussing the zero-shot and few-shot results; the corresponding tables are labeled Table III and Table IV, respectively.","section":"Section IV-D1"},{"comment":"The ablation results for the VAE-GAN network are said to be shown in 'Figure 3' at the beginning of Section IV-E1, but the referenced figure number appears to be incorrect; the paper should refer to the actual figure or table number containing those results.","section":"Section IV-E1"},{"comment":"The notation in the GAN objectives is inconsistent: Eq. (2) uses Dv(ev, a; θDv, θGv), but ev appears to be generated by Gv, so the argument should reflect the generator parameters. Also, the gradient penalty term is written as if it is evaluated with respect to θDv, but the expectation over training examples is not specified clearly; standard WGAN-GP notation should be used for readability.","section":"Equation (2) and (4)"},{"comment":"The definition of the contrastive loss in Eq. (13)-(14) is ambiguous: the superscript of u∗i includes the modal indicator and the subscript i, but the summation in Eq. (13) is not fully specified. Clarifying the index sets and the exact form of P (u∗i_i) would improve reproducibility.","section":"Section III-E2"},{"comment":"The row 'Ours(1-shot)' contains results for 1-, 3-, and 5-shot; this should be relabeled to 'Ours' with separate shot rows for clarity, and the CLIP row should clearly indicate that the CLIP baseline is not trained in the few-shot setting (as the paper already notes with '-' for sample information).","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and contains a reasonable architecture, but the current version is not yet ready for publication. The most serious issue is the contradiction between the zero-shot protocol (Section III-A) and the training description in Section IV-C, which casts doubt on the headline zero-shot results. In addition, the experimental reporting lacks code, seeds, and variance estimates, and the claimed improvements are not consistently quantified. I suggest requesting the authors to (1) clarify whether any target-domain real samples are used in stage 1 and, if so, rerun the zero-shot experiments under the strict protocol, (2) add distributional evidence for the pseudo-features, and (3) improve the clarity of the tables and equations. If the authors can provide these, the paper could be a moderate contribution to the field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"James,\n\nShort version: the few-shot architecture is a reasonable engineering combination, but the zero-shot claim as written doesn't survive contact with the paper's own implementation details, and the empirical evidence is thinner than the margins suggest.\n\nWhat's new: combining F-VAEGAN-D2-style feature generation with CLIP features and a gated residual fusion for cross-modal retrieval. The gate-residual idea is the most interesting piece—it directly targets a real failure mode where mapping CLIP features into a common space hurts more than it helps in low-shot regimes. The ablations back this up: removing the gate costs a lot, and removing the VAE also hurts. The gains over the old non-CLIP baselines are consistent across four datasets. That part is credible as an incremental contribution.\n\nWhere it falls apart: Section III-A defines zero-shot as training only on source data. Section IV-C says the first training phase uses \"source domain data and a small amount of target domain data.\" Those can't both be true for the zero-shot experiments reported in Table III. If real target features were used in stage 1, the comparison to frozen CLIP is not zero-shot and the up-to-7.9% margin is not what it claims. If the phrase means only target class embeddings, the paper needs to say so, and it still owes the reader evidence that generated target features actually resemble real target features—there are no distributional diagnostics. This is the load-bearing issue, and it's in the paper, not an artifact of the review.\n\nOther softness: the abstract's 7%-15% doesn't match the 7.9% in the contributions, and the per-dataset numbers in Section IV-D1 are internally inconsistent (Wikipedia gain reported as 2.31 while the table shows 0.024 MAP difference). No code, no seeds, no error bars; the CLIP baseline is frozen while the proposed method trains on top of CLIP features, so the comparison is uneven. These are fixable with a revision, but they matter.\n\nWho should read it: people working on few-shot cross-modal retrieval will find the gate-residual fusion worth a look, and the setup is clearly described enough to rebuild. But the paper needs a serious revision, and the authors should release code.\n\nRecommendation: send it to peer review—the problem is real and the architecture is sensible—but reviewers should push hard on the zero-shot protocol and the missing uncertainty estimates. As it stands, I would not cite the zero-shot numbers.","headline":"Plausible VAE-GAN + CLIP fusion for few-shot CMR, but the zero-shot headline is undercut by an internal contradiction about using target-domain data in stage 1.","tokens_in":37276,"tokens_out":3139,"would_cite":false,"duration_ms":27560,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLEX-CLIP claims that generating synthetic target features and gating them with original CLIP features lets few-shot and zero-shot cross-modal retrieval beat the CLIP baseline by up to 7.9%.","keywords":["cross-modal retrieval","few-shot learning","zero-shot learning","CLIP","feature generation","VAE-GAN","gate residual network","data imbalance"],"falsifier":"A concrete check is to train the VAE-GAN on source features, generate pseudo-features for the target classes, and compute the distribution distance (for example, Fréchet distance or maximum mean discrepancy) between generated and real target features in the CLIP feature space; if that distance does not correlate with the reported retrieval improvements, or if replacing the generated pseudo-features with random noise from the same class embeddings yields the same mAP, the central claim that generation is the cause of the gains would be falsified.","tokens_in":36303,"feed_emoji":"🎯","tokens_out":3214,"duration_ms":32397,"temperature":0.7,"pith_summary":"The paper tries to establish that the two main failures of CLIP-based few-shot cross-modal retrieval, target-domain feature degradation and extreme data imbalance, can both be fixed at the feature level. It proposes FLEX-CLIP, a two-stage method: first a composite VAE-GAN per modality generates pseudo target-domain samples from class embeddings, and second a gate residual network selectively fuses the original CLIP features with the projected features before retrieval. On four benchmark datasets across 0, 1, 3, and 5-shot settings, the method reports consistent gains over strong baselines and, notably, is the only model in its comparisons that surpasses the frozen CLIP features in zero-shot retrieval. A sympathetic reader would take this as evidence that feature-level generation plus selective fusion is a viable recipe for extending pretrained vision-language models to unseen classes.","feed_headline":"Synthetic CLIP features lift few-shot retrieval by up to 7.9%","feed_subtitle":"FLEX-CLIP generates pseudo target samples and gates original CLIP features to beat strong baselines on four datasets.","key_machinery":"The central mechanism is the composite multimodal VAE-GAN plus the gate residual network. Each modality (image and text) has an encoder E, a generator G that serves as both GAN generator and VAE decoder, and a class-conditional discriminator D. The VAE loss makes G reconstruct real features from latent codes, the GAN loss makes G produce plausible pseudo features from class embeddings, and the same G is shared, so the generator learns the real feature distribution from source data and transfers that to the target classes. The gate residual network outputs a coefficient vector g from the concatenation of the original CLIP feature and the projected feature, and computes the final feature as g times the projected feature plus (1-g) times the original feature, allowing an adaptive per-dimension compromise between pretrained semantics and task-specific projection.","core_discovery":"The central claim is that FLEX-CLIP, by decoupling multimodal feature generation from common-space projection, solves the data imbalance and feature degradation problems that limit CLIP in x-shot cross-modal retrieval. In the generation stage, a composite VAE-GAN per modality uses the generator both as a GAN decoder (producing pseudo target samples from class attributes and noise) and as a VAE decoder (reconstructing real features from latent variables), so the generated distribution is constrained by both reconstruction and adversarial losses. In the projection stage, a gate network computes a per-dimension fusion coefficient between the raw CLIP feature and the projected feature, so the final common-space representation can retain useful pretrained semantics when the projection is unreliable. The paper reports that this two-stage design improves average mean average precision over state-of-the-art methods by 7% to 15% on four datasets, and in zero-shot settings outperforms the original CLIP features by margins from 2.31 to 7.90.","pith_inferences":["The paper's ablation shows that removing the VAE-GAN still leaves the model above CLIP, suggesting that the gate residual network alone may be a lightweight fix for feature degradation; one extension would be to test the gate on other frozen encoders and other downstream tasks.","Because the generation stage operates on precomputed CLIP features, the approach could be transferred to other vision-language pretraining models or to other feature spaces, provided the class embeddings align with those features.","An indirect consequence is that the value of the generated pseudo-features depends on how well the VAE-GAN captures the source distribution; a testable extension is to measure the distribution distance between generated and real target features and correlate it with retrieval gains."],"forward_implications":["If pseudo-feature generation works as claimed, few-shot and zero-shot cross-modal retrieval no longer need to wait for large labeled target-domain data; the target class embeddings alone can synthesize training samples.","If the gate residual network works as claimed, fine-tuning CLIP for a new task does not have to forget the pretrained representations, since the gate can down-weight the projected features when they are harmful.","The two-stage separation of generation from projection implies that each stage can be improved or swapped independently, so better generative models or better projectors could be plugged in without retraining the whole pipeline.","The reported gains on Pascal Sentence and NUS-WIDE suggest the method is not dataset-specific, but the noise sensitivity observed on larger datasets indicates that the quality of the few real samples still matters."],"supporting_citations":[{"why":"Supplies the CLIP image and text encoders whose features are enhanced, and defines the original-feature baseline that FLEX-CLIP must surpass.","marker":"[22]"},{"why":"The CFSA method provides the coupled Wasserstein GAN approach to feature synthesis that FLEX-CLIP extends with a VAE component.","marker":"[5]"},{"why":"JFSE is both a baseline and the source of the joint feature-synthesis and embedding idea that FLEX-CLIP explicitly separates into two stages.","marker":"[13]"},{"why":"MDVAE is a VAE-based zero-shot cross-modal retrieval baseline and motivates the use of VAE to model feature distributions.","marker":"[12]"},{"why":"AAEGAN combines autoencoder and GAN for zero-shot cross-modal retrieval, the direct precursor of the composite VAE-GAN architecture.","marker":"[11]"},{"why":"f-VAEGAN-D2 provides the feature-generating framework for any-shot learning that inspires the pseudo-sample generation from class embeddings.","marker":"[17]"},{"why":"DADN is a zero-shot cross-media dual adversarial network baseline that FLEX-CLIP compares against and improves upon.","marker":"[7]"}],"fun_headline_variants":["FLEX-CLIP: Synthetic features boost few-shot retrieval by up to 15%","Gate and generate: FLEX-CLIP improves CLIP cross-modal retrieval","FLEX-CLIP uses VAE-GAN and gates to fix CLIP's few-shot gaps","Pseudo samples and gate fusion: FLEX-CLIP lifts few-shot CMR","Synthetic data and gating: FLEX-CLIP enhances CLIP for x-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a generative model trained only on source-domain features and class embeddings can produce target-domain pseudo-features whose distribution is close enough to the real target features that training the projection network on them improves retrieval, and this is especially fragile in the zero-shot case where no real target samples are seen at all.","fun_headline_variants_meta":{"raw":{"variants":["FLEX-CLIP: Synthetic features boost few-shot retrieval by up to 15%","Gate and generate: FLEX-CLIP improves CLIP cross-modal retrieval","FLEX-CLIP uses VAE-GAN and gates to fix CLIP's few-shot gaps","Pseudo samples and gate fusion: FLEX-CLIP lifts few-shot CMR","Synthetic data and gating: FLEX-CLIP enhances CLIP for x-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3581,"prompt_tokens":951,"completion_tokens":2630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2516}},"tokens_in":567,"tokens_out":2630,"duration_ms":17494,"temperature":1.0,"reasoning_tokens":2516,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:06:59.262780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to train the VAE-GAN on source features, generate pseudo-features for the target classes, and compute the distribution distance (for example, Fréchet distance or maximum mean discrepancy) between generated and real target features in the CLIP feature space; if that distance does not correlate with the reported retrieval improvements, or if replacing the generated pseudo-features with random noise from the same class embeddings yields the same mAP, the central claim that generation is the cause of the gains would be falsified.","supporting_citations":[{"cited_title":"Correlated features synthesis and alignment for zero-shot cross-modal retrieval,","cited_arxiv_id":null,"evidence_quote":"The CFSA method provides the coupled Wasserstein GAN approach to feature synthesis that FLEX-CLIP extends with a VAE component."},{"cited_title":"Joint feature syn- thesis and embedding: Adversarial cross-modal retrieval revisited,","cited_arxiv_id":null,"evidence_quote":"JFSE is both a baseline and the source of the joint feature-synthesis and embedding idea that FLEX-CLIP explicitly separates into two stages."},{"cited_title":"Multimodal disentanglement variational autoencoders for zero-shot cross-modal re- trieval,","cited_arxiv_id":null,"evidence_quote":"MDVAE is a VAE-based zero-shot cross-modal retrieval baseline and motivates the use of VAE to model feature distributions."},{"cited_title":"Zero- shot cross-modal retrieval by assembling autoencoder and generative adversarial network,","cited_arxiv_id":null,"evidence_quote":"AAEGAN combines autoencoder and GAN for zero-shot cross-modal retrieval, the direct precursor of the composite VAE-GAN architecture."},{"cited_title":"F-vaegan-d2: A feature generating framework for any-shot learning,","cited_arxiv_id":null,"evidence_quote":"f-VAEGAN-D2 provides the feature-generating framework for any-shot learning that inspires the pseudo-sample generation from class embeddings."},{"cited_title":"Zero-shot cross-media embedding learning with dual adversarial distribution network,","cited_arxiv_id":null,"evidence_quote":"DADN is a zero-shot cross-media dual adversarial network baseline that FLEX-CLIP compares against and improves upon."}],"review_version":1}