{"id":"40276919-c073-48a2-8419-c3c6720bf768","arxiv_id":"2412.17219","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DIG-ZSL learns a per-class token that steers Stable Diffusion to generate discriminative images for unseen classes, then trains a ZSL classifier on those images.","lead":"This paper generates images of unseen classes with a text-to-image diffusion model, using learned discriminative tokens to make the images useful for zero-shot classification. The authors report large gains over prior methods that use nonhuman-annotated semantic prototypes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot claim rests on Stable Diffusion and CLIP not having been pretrained on the benchmark unseen classes; the paper neither establishes nor discusses this, so the reported gains may reflect memorization rather than transfer.","rationale":"The reader's weakest assumption identified exactly the concern I consider most load-bearing: Stable Diffusion and CLIP may have been pretrained on the benchmark unseen classes, so the reported accuracies may reflect memorization rather than zero-shot transfer. I agree with this assessment because the paper's entire argument depends on the separation between seen and unseen classes, and the method relies on frozen web-scale models at two critical points: generating the unseen-class images and computing the features used for classification. If those models have seen the test classes, the central claim is not merely overstated; the evaluation no longer measures zero-shot learning. The missing vanilla-SD baseline is a related but secondary issue: it would reveal whether the DCT optimization is responsible for the gains, but the contamination concern would remain even if the DCT stage were shown to help. I am not accusing the authors of dishonesty; this is a standard but serious correctness risk when adapting pretrained generative and vision-language models to benchmarks whose classes likely overlap with web-scale training data. The concrete test I propose directly separates the contribution of the DCT from the prior knowledge already present in Stable Diffusion and CLIP, and it is feasible with the same experimental infrastructure. I therefore recommend keeping the reader's CONDITIONAL verdict: the paper should not be accepted as a clean zero-shot result until either the vanilla-SD control is reported or a credible training-data overlap audit is provided.","tokens_in":15684,"tokens_out":7357,"duration_ms":73622,"concrete_test":"Run the full DIG-ZSL classifier pipeline with the DCT stage removed: generate the same 100 images per unseen class using vanilla Stable Diffusion with only the class-name prompt 'A photo of a [name]', train the same MLP classifier on these images plus real seen-class images, and evaluate under the same CZSL/GZSL protocols and lambda values. If this vanilla-SD baseline reaches accuracy comparable to DIG-ZSL (within a few points), the reported gains are not attributable to the proposed discriminative class tokens, and the high absolute accuracy is consistent with SD/CLIP having memorized the unseen classes. If instead vanilla SD is much worse, the DCT mechanism contributes something real, though an independent training-data overlap audit would still be needed to fully clear the contamination concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DIG-ZSL's central claim is that a classifier trained on images generated for unseen classes can recognize real unseen classes. For this to be a genuine zero-shot result, the models used to generate and evaluate those images must not have been trained on the unseen classes. The pipeline in §4.1 uses Stable Diffusion v2.1 for generation and CLIP ViT-B/16 both to define semantic prototypes and as the visual backbone of the CDM and final classifier. Both are trained on web-scale image-text data (SD v2.1 on LAION-5B), which very likely contains images and captions for AWA2, CUB, FLO, and SUN classes. If so, the generated images and the CLIP features used for classification can encode knowledge of the unseen classes, and the reported improvements (e.g., 90.1 vs 77.3 on AWA2 in Table 1) may be due to memorization rather than to the DCT/CDM mechanism. This is not merely a theoretical worry: the paper does not audit training-set overlap, does not report a vanilla-SD control under the same classifier, and never discusses the contamination risk in the main text or appendix.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DIG-ZSL, a zero-shot learning method that synthesizes images for unseen classes with Stable Diffusion. In the first stage, a Category Discrimination Model (CDM), composed of a frozen CLIP visual encoder and a trainable MLP, is trained on seen classes to align visual features with CLIP text embeddings. In the second stage, for each unseen class a discriminative class token (DCT) is optimized so that images generated with the prompt \"A photo of S* [name]\" maximize the CDM-based cosine similarity to that class's CLIP prototype. In the third stage, a classifier is trained on real seen images plus generated unseen images and evaluated under both conventional and generalized ZSL on AWA2, CUB, FLO, and SUN. The paper reports large improvements over nonhuman-annotated semantic prototype baselines and competitive results against human-annotated attribute methods.","tokens_in":15927,"tokens_out":9517,"duration_ms":89082,"significance":"If the results reflect genuine zero-shot transfer, the paper makes a useful contribution: it demonstrates an interpretable image-generation route to ZSL, removes the need for human-annotated attributes, and provides extensive experiments, visualizations, and hyperparameter analyses. The CDM-guided DCT idea is simple and plausible. However, the significance is currently conditional on two unresolved issues: possible contamination of the web-scale pretrained SD/CLIP models with the benchmark unseen classes, and the absence of a vanilla-SD classification control. The paper also omits details of the DCT optimization and of the validation protocol. Until these are addressed, the headline accuracies cannot be interpreted as evidence of zero-shot generalization. No code is released, so reproducibility rests entirely on the written description.","major_comments":[{"comment":"","section":"§4.1, Implementation Details, with §3.2–§3.4"},{"comment":"","section":"§4.3, Table 4"},{"comment":"","section":"§3.3 and Appendix A"},{"comment":"","section":"§4.1 and §4.5"}],"minor_comments":[{"comment":"The heading \"Stable Difusion\" contains a typo; it should read \"Stable Diffusion.\"","section":"§3.1"},{"comment":"The dataset name is inconsistent: \"AW A2\" appears in the text while \"AWA2\" appears in tables; please use one canonical form.","section":"§1, §4.1, Table 3"},{"comment":"In the \"Supervised Classifier Training\" paragraph, the text says the default setting is the same as CDM \"as shown in Tab. 7,\" but Tab. 7 reports DCT learning settings; this cross-reference appears to be wrong.","section":"Appendix A"},{"comment":"Table 4 reports a single FID value per method without seeds or confidence intervals; given the small margins, the comparison would be more informative with multiple runs or error bars.","section":"§4.3, Table 4"},{"comment":"The phrase \"as indicated in gray font\" cannot be verified in the text version of Table 2; please use a dedicated column or note to identify the large vision-language baselines.","section":"§4.2, Table 2"},{"comment":"Table 3 omits FLO without explanation; please either include FLO results or note why it is excluded.","section":"§4.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The contamination concern is the gating issue. If an overlap audit shows that the benchmark unseen classes appear in LAION-5B or CLIP training data, the paper's central zero-shot claim would not be supported. The missing vanilla-SD classification control is equally necessary to establish that DCT optimization contributes to accuracy. Both issues are addressable experimentally, so I would not reject outright, but I would not accept without these additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: DIG-ZSL takes the discriminative class token (DCT) idea from Schwartz et al. and repurposes it for zero-shot learning, training the guidance model only on seen classes. The reported gains over nonhuman-annotated prototype baselines are large, and the idea of generating interpretable images rather than features is genuinely useful. But the evaluation skips the one baseline that would make the claim credible, and it ignores the elephant in the room: Stable Diffusion and CLIP were trained on web-scale data that almost certainly includes the 'unseen' classes. That has to be addressed before I'd trust the numbers.\n\nWhat's actually new: combining DCT optimization with a seen-only category discrimination model, and applying it to ZSL. That is a legitimate extension, not a copy. The paper is clearly written, the comparisons to other nonhuman-annotated methods are thorough, and the visualizations show that DCTs can improve text-image alignment (e.g., the yellow bill of the Red-legged Kittiwake). The t-SNE of token embeddings and the sensitivity analysis on Ngen and λ are useful.\n\nSoft spots, in order of severity:\n\n1. No vanilla SD control. The paper never trains the same ZSL classifier on images generated by plain Stable Diffusion with just 'A photo of a [name]' prompts. The only comparison is FID, and the differences (19.7 vs 19.6, 50.5 vs 49.9) are negligible. For all we know, the accuracy gains come almost entirely from using high-quality SD images, not from the DCT. This is a one-hour experiment and it's the missing cornerstone.\n\n2. Contamination risk. SD v2.1 and CLIP ViT-B/16 are trained on LAION-5B and similar. AWA2, CUB, FLO, and SUN are standard datasets that have been on the internet for years. The paper neither audits overlap nor discusses it. If the generative model and the feature extractor have seen the unseen classes, the term 'zero-shot' is doing a lot of work. The authors need to either show an overlap analysis, use a model with known exclusion, or at least acknowledge the limitation and test robustness (e.g., generating with a smaller diffusion model not pretrained on those classes, or checking retrieval of generated images).\n\n3. Per-dataset hyperparameters. γ and λ are hand-set per dataset, and the paper doesn't justify them on a validation split. The sensitivity analysis only shows λ on CUB. Since λ directly calibrates the seen/unseen bias, this is a source of selection bias. The early-stopping threshold γ also affects results; the appendix shows γ=0.2 to 0.8 changes images substantially, but there's no analysis of accuracy vs γ.\n\n4. Method details. The DCT optimization is described at a high level, but the gradient flow through the diffusion sampling process is not explained. How exactly does the cross-entropy loss propagate to e*? This matters because the forward pass involves stochastic sampling, and the paper doesn't say whether they use a score distillation-like trick, a differentiable sampler, or something else. A reviewer will need this.\n\nNone of these are fatal on their own, but together they mean the central claim—that a classifier trained on generated discriminative images recognizes real unseen classes—is not yet established. The fix is straightforward: add the vanilla SD baseline, do a contamination audit, and provide validation-set-based hyperparameter selection.\n\nWho should read this: anyone working on generative ZSL or using diffusion models for data augmentation. It deserves a serious referee, but the referee should push for the missing experiments before publication. I'd send it to review, not desk reject, but I'd expect major revisions.\n\nBest,\n[Your name]","headline":"Strong new idea for image-based ZSL, but missing vanilla-SD control and contamination audit leave the headline claim unproven.","tokens_in":16469,"tokens_out":3500,"would_cite":false,"duration_ms":28119,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A diffusion-based pipeline that generates discriminative images for unseen classes can train a zero-shot classifier, surpassing prior nonhuman-annotated semantic prototype methods by a large margin on four benchmarks.","keywords":["zero-shot learning","generative zero-shot learning","text-to-image diffusion models","discriminative class tokens","semantic prototypes","generalized zero-shot learning"],"falsifier":"Run the exact DIG-ZSL protocol on a set of newly created classes that provably did not exist when Stable Diffusion and CLIP were trained, verify their absence by retrieval-style overlap checks, and measure Top-1 accuracy; if accuracy collapses toward chance level on such held-out classes while staying high on AWA2, CUB, FLO, and SUN, the benchmark result would be explained by pretraining leakage rather than by the proposed mechanism.","tokens_in":15468,"feed_emoji":"🖼️","tokens_out":7218,"duration_ms":61277,"temperature":0.7,"pith_summary":"Zero-shot learning normally synthesizes feature vectors for unseen classes from human-annotated attributes. The paper claims this step can be replaced by generating actual images with a text-to-image diffusion model, using only class names as semantic information. To make the images useful for classification, the paper learns a discriminative class token for each unseen class under the guidance of a category discrimination model trained on seen classes. The paper reports that a classifier trained on these generated images plus real seen images sets new state-of-the-art results for nonhuman-annotated semantic prototypes, with an average Top-1 improvement of 24.2% over prior methods, and matches or exceeds methods that use human-annotated attributes on three of four benchmarks. If correct, the result removes the main scalability bottleneck of generative zero-shot learning: no expert attribute annotation is needed for new classes.","feed_headline":"Diffusion images train zero-shot classifiers that beat prior methods","feed_subtitle":"Using only class names as semantic prototypes, DIG-ZSL tops nonhuman-annotated ZSL baselines by 24.2% average accuracy.","key_machinery":"The carrying object is the discriminative class token: a single learned embedding vector added to the tokenizer vocabulary and placed into the diffusion prompt between 'A photo of' and the class name. The token is optimized so that images generated by Stable Diffusion receive high cosine similarity under the category discrimination model to the correct unseen class's CLIP text prototype and low similarity to other unseen classes; an early-stopping threshold on classification accuracy prevents over-training. The category discrimination model provides the only learning signal from seen data, so the whole pipeline needs no human-annotated attributes. The same backbone features are reused when training the final classifier.","core_discovery":"The central claim is that zero-shot recognition can be driven by image generation: instead of learning a conditional generator of visual features, one can prompt a frozen text-to-image diffusion model with a learned discriminative class token to synthesize photorealistic training images for each unseen class. The token is inserted into the prompt 'A photo of S* [name]' and optimized under the guidance of a category discrimination model, a frozen vision backbone with a trainable projection trained only on seen classes, using a cross-entropy loss on cosine similarities between the generated image's feature and the CLIP text prototypes of the unseen classes. The generated images are then combined with real seen data to train a standard classifier for both conventional and generalized zero-shot settings. The paper demonstrates the claim on AWA2, CUB, FLO, and SUN, reporting substantial improvements over nonhuman-annotated semantic prototype baselines and accuracy competitive with human-annotated attribute methods; it also shows that fine-tuning the category discrimination model on seen data raises accuracy further and that generated images have lower FID than plain Stable Diffusion outputs.","pith_inferences":["Inference: Because the method's benchmark numbers could be inflated if Stable Diffusion or CLIP encountered the unseen classes during web-scale pretraining, the zero-shot claim would be strengthened by repeating the protocol on genuinely novel classes invented after the training cutoff.","Inference: The token-learning step is decoupled from the classifier, so the same discriminative class tokens could be reused to generate images at any resolution or count later, or to feed other downstream tasks such as retrieval and captioning.","Inference: The category discrimination model is modular; replacing the vision backbone and projection with any model that can score generated images against unseen-class semantic prototypes, trained only on seen data, would likely yield similar behavior.","Inference: A few-shot extension is natural: a handful of real images from a new class could refine its token, potentially closing the remaining gap on fine-grained datasets like CUB where the paper reports lower accuracy."],"forward_implications":["If the central claim holds, generative zero-shot learning no longer needs human-annotated attributes: class names alone suffice, so new categories can be added without expert annotation cost.","Generated images provide interpretable evidence of what the model believes an unseen class looks like, which feature-synthesis methods do not offer.","The same pipeline applies to both conventional and generalized zero-shot settings; on the reported benchmarks, it surpasses previous nonhuman-annotated semantic prototype methods and remains competitive with human-annotated ones.","The number of generated images per class is a controllable resource; accuracy rises with more samples and saturates around 100 per class, so users can trade generation cost against accuracy.","Improving the category discrimination model, for example by fine-tuning on seen classes, is a direct lever that further boosts the final zero-shot accuracy by 1.2 to 5.1 percentage points on the tested datasets."],"supporting_citations":[{"why":"Supplies the Stable Diffusion text-to-image model used to generate the training images for unseen classes.","marker":"[41]"},{"why":"Supplies the CLIP text encoder whose class-name embeddings serve as semantic prototypes and whose ViT-B/16 backbone is used in the category discrimination model.","marker":"[38]"},{"why":"Contributes the discriminative class token optimization idea that DIG-ZSL adapts to the zero-shot setting with a seen-only guidance model.","marker":"[45]"},{"why":"Defines the benchmark datasets, seen/unseen splits, and evaluation protocols used in all comparisons.","marker":"[53]"},{"why":"I2DFormer is one of the nonhuman-annotated semantic prototype baselines that DIG-ZSL must beat.","marker":"[31]"},{"why":"I2DFormer+ is the previous state-of-the-art nonhuman-annotated method in the conventional zero-shot comparison.","marker":"[33]"},{"why":"I2MVFormer-LLM is the previous state-of-the-art nonhuman-annotated method in the generalized zero-shot comparison.","marker":"[32]"},{"why":"ZeroDiff is the prior diffusion-based zero-shot work that DIG-ZSL contrasts with by moving from feature generation to image generation.","marker":"[60]"}],"fun_headline_variants":["Diffusion images train zero-shot classifiers without human attributes","Discriminative diffusion images for zero-shot recognition","Image generation outperforms feature synthesis in zero-shot learning","Zero-shot learning with generated images matches attribute methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on Stable Diffusion and CLIP not already having seen images of the benchmark 'unseen' classes during pretraining; if those classes appear in the training data, the generated images and the CLIP features carry class information, and the reported accuracy measures memorization rather than zero-shot transfer.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion images train zero-shot classifiers without human attributes","Discriminative diffusion images for zero-shot recognition","Image generation outperforms feature synthesis in zero-shot learning","Zero-shot learning with generated images matches attribute methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001111,"raw_usage":{"total_tokens":4661,"prompt_tokens":1008,"completion_tokens":3653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":3592}},"tokens_in":624,"tokens_out":3653,"duration_ms":25502,"temperature":1.0,"reasoning_tokens":3592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:42:04.151875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact DIG-ZSL protocol on a set of newly created classes that provably did not exist when Stable Diffusion and CLIP were trained, verify their absence by retrieval-style overlap checks, and measure Top-1 accuracy; if accuracy collapses toward chance level on such held-out classes while staying high on AWA2, CUB, FLO, and SUN, the benchmark result would be explained by pretraining leakage rather than by the proposed mechanism.","supporting_citations":[{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the Stable Diffusion text-to-image model used to generate the training images for unseen classes."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP text encoder whose class-name embeddings serve as semantic prototypes and whose ViT-B/16 backbone is used in the category discrimination model."},{"cited_title":"Discriminative class tokens for text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Contributes the discriminative class token optimization idea that DIG-ZSL adapts to the zero-shot setting with a seen-only guidance model."},{"cited_title":"Zero-shot learning—a comprehensive eval- uation of the good, the bad and the ugly","cited_arxiv_id":null,"evidence_quote":"Defines the benchmark datasets, seen/unseen splits, and evaluation protocols used in all comparisons."},{"cited_title":"I2dformer: Learning image to document attention for zero-shot image classification","cited_arxiv_id":null,"evidence_quote":"I2DFormer is one of the nonhuman-annotated semantic prototype baselines that DIG-ZSL must beat."},{"cited_title":"I2dformer+: Learning image to doc- ument summary attention for zero-shot image classification","cited_arxiv_id":null,"evidence_quote":"I2DFormer+ is the previous state-of-the-art nonhuman-annotated method in the conventional zero-shot comparison."},{"cited_title":"I2mvformer: Large language model generated multi-view document supervi- sion for zero-shot image classification","cited_arxiv_id":null,"evidence_quote":"I2MVFormer-LLM is the previous state-of-the-art nonhuman-annotated method in the generalized zero-shot comparison."}],"review_version":1}