{"id":"f941b03d-c111-4bf7-80e4-791a5a8ec4c3","arxiv_id":"2506.23822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LaZSL aligns local image patches with LLM-generated attribute descriptions using optimal transport, achieving slightly higher average zero-shot accuracy than prior interpretable CLIP baselines on nine benchmarks.","lead":"LaZSL is a training-free method that improves zero-shot image classification by aligning random image crops with text attributes via optimal transport. It offers a more interpretable alternative to standard CLIP scoring, though the reported gains are modest and the evaluation has some methodological caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OT plan may be near-uniform: λ is unreported and Fig. 3 attribute scores are flat, so accuracy gains could come from multi-crop ensembling rather than local semantic alignment.","rationale":"The reader's weakest assumption focuses on whether random crops contain the evidence described by attributes. I agree that this is the key empirical gap, but I would sharpen it into a mechanism-level concern: even if the crops are informative, the OT plan could be trivial because the entropic regularization λ in Eq. 14 is never reported, and the qualitative results in Fig. 3 show near-uniform attribute contributions. If the OT plan is near-uniform, Eq. 15 is essentially a weighted average of crop-attribute similarities, and the claimed advantage of 'optimal transport local alignment' over DCLIP-style averaging disappears. The paper's ablations add OT, then vision selection, then hybrid; they never remove the semantic content of the attributes or replace OT with a simple average, so the unique contribution of the proposed alignment cannot be identified. This concern is more load-bearing than the hyperparameter tuning on test benchmarks because if the mechanism reduces to ensembling, the central technical novelty is void even if the numbers are reproducible. The proposed concrete test is directly implementable with the released code: sweep λ to see if the algorithm is sensitive to the unstated temperature, measure the OT plan's entropy, and run a class-name-only control. If the class-name-only control performs nearly as well, the LLM-generated attributes and their OT alignment are not the active ingredient. I do not think the concern warrants rejection because the method may still work for an appropriate λ and the accuracy gains could be real; however, the paper must report λ and provide the control before the central claim can be accepted. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":13996,"tokens_out":6283,"duration_ms":74048,"concrete_test":"Using the released code, run ViT-B/16 on CUB and ImageNet with: (1) λ swept over {0.001, 0.01, 0.1, 1, 10} (reporting any default), measuring top-1 accuracy and the entropy of the OT plan T; and (2) a control that replaces each class's attribute set with the class name only (M=1) while keeping crops, vision selection, and hybrid cost identical. If accuracy varies by less than 1% across λ or T is near-uniform, or if the class-name control loses less than 1% relative to LaZSL, then OT-based local alignment is not the source of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LaZSL's central claim is that the OT plan in Eq. 11 delivers meaningful local visual-semantic alignment. This requires that (i) the filtered crops actually contain attribute-relevant evidence and (ii) the Sinkhorn plan is non-trivial. Neither is verified. The entropic regularization λ in Eq. 14 and the iteration count k are never specified, so the OT plan's sharpness is uncontrolled. If λ is large, T collapses toward the product of uniform margins, and Eq. 15 reduces to a multi-crop average of attribute similarities, which is indistinguishable from DCLIP with random-crop ensembling. The qualitative evidence in Fig. 3 is consistent with this worry: all attribute scores are nearly identical (≈0.067), suggesting the plan is dispersive. The ablations in Table 3 only add components cumulatively; there is no control that replaces attributes with class names or OT with averaging, so the causal role of local semantic alignment is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LaZSL, a training-free interpretable zero-shot learning method built on CLIP. It constructs a semantic set of LLM-generated attributes per class and a visual set of random multi-scale crops per image, filters crops by similarity to the global image embedding, computes a Sinkhorn optimal-transport plan between filtered local visual features and attribute features, and scores each class with a hybrid local-global Frobenius inner product. The authors report accuracy gains over DCLIP, WaffleCLIP, CuPL, and several prompt-learning baselines on five classification benchmarks and four ImageNet variants, plus qualitative examples intended to show attribute-based interpretability.","tokens_in":14179,"tokens_out":4795,"duration_ms":56970,"significance":"If the reported results are taken at face value, the paper offers a useful training-free way to obtain attribute-level explanations from CLIP while improving average accuracy, and it ships code and a relatively broad evaluation across nine datasets. The core derivation is straightforward and the component ablations are directionally consistent. However, the evaluation currently has load-bearing weaknesses: hyperparameters are selected on the same test benchmarks used in the headline tables, the entropic OT parameters are never reported, the ablations do not isolate local semantic alignment from multi-crop ensembling, and there is no direct verification that selected crops contain the evidence described by the attributes. These issues prevent the paper from fully establishing its central claim that locally-aligned OT, rather than ensembling or test-set tuning, drives the improvements.","major_comments":[{"comment":"Hyperparameter selection is performed on the same benchmarks used for the headline results. Figure 5 chooses α and θ by evaluating accuracy on CUB and ImageNet, and those same two datasets appear in Tables 1 and 2 with the final accuracy reported. This means the gains over DCLIP (e.g., +3.8 on CUB in Table 1) are partly a function of test-set tuning, so the central accuracy claim is not evaluated cleanly. Please fix α, θ, N, and β to predetermined values, tune on held-out validation splits, or explicitly state which datasets were used for tuning; if tuning was done on the test benchmarks, the comparisons should be re-run with settings selected without access to those test labels.","section":"Section 4.5, Tables 1–2"},{"comment":"The entropic regularization λ and the number of Sinkhorn iterations k are never reported. Equation (14) defines M = exp(−C*/λ), so if λ is large the OT plan T collapses toward the product of the uniform marginals and Eq. (15) becomes a multi-crop average of attribute similarities. In that regime the method is indistinguishable from random-crop ensembling with global hybrid features. The qualitative evidence in Fig. 3 is consistent with this worry: the displayed attribute scores are nearly identical (about 0.067). Please report λ and k, measure the sharpness of T (e.g., mean maximum entry or the effective number of matches), and add an ablation that replaces the OT plan with simple averaging over the same filtered crops.","section":"Section 3.2, Eq. (14)"},{"comment":"The ablation study only adds components cumulatively relative to DCLIP, so it does not establish the causal role of local visual-semantic alignment. There is no control that keeps the OT pipeline but replaces LLM attributes with class-name embeddings, and no control that replaces OT with average pooling over the same filtered crops. Without these controls, the gains in Table 3 could come from multi-crop ensembling or the hybrid global term rather than from aligning local regions with attributes. Please add the missing controls, including a uniform-plan variant of Eq. (15).","section":"Table 3, Ablation Study"},{"comment":"The method assumes that random multi-scale crops, after filtering by similarity to the global image embedding, contain the local evidence described by the LLM attributes. This assumption is not directly verified. Please provide quantitative evidence that selected regions correspond to the attributes used for scoring, for example by measuring region-attribute retrieval precision on a subset with part annotations or by visualizing which crops receive high OT mass for a given attribute. Without such verification, the OT plan may be adding noise rather than signal, which would undermine the interpretability and alignment claims.","section":"Section 3.1, Eqs. (6)–(8)"},{"comment":"Tables 1 and 2 report single accuracy numbers with no error bars or repeated trials, while several claimed improvements are small (e.g., +0.2 and +0.5 points on Food101 and Oxford Pets in Table 1) and the largest domain-generalization gain is concentrated in ImageNet-A (+6.0 over CuPL in Table 2). Please report mean ± std over at least three random-crop or random-seed runs. In addition, the provenance of the SaLS baseline in Table 2 should be clarified: reference [32] in the bibliography is a duplicate of reference [31] and does not correspond to the SaLS method cited in the table.","section":"Tables 1–2"}],"minor_comments":[{"comment":"The notation for C*_i and sim*_i mixes vector and matrix expressions; please clarify the dimensions and broadcasting of the global term pg⊤Qy so the hybrid cost and similarity are unambiguous.","section":"Section 3.2, Eqs. (10) and (16)"},{"comment":"There are typos in the figure: 'LAZSL' should be 'LaZSL' and 'bely' should be 'belly'.","section":"Figure 1"},{"comment":"The footnote describing the training and attribute markers is incomplete: the sentence 'Methods take the source data for training marked with' cuts off, and the meaning of the quoted and percent symbols should be stated explicitly.","section":"Table 2, footnote"},{"comment":"The Limitations paragraph correctly notes dependence on LLM-generated attribute quality, but the paper does not quantify robustness to attribute noise. Please add an analysis that perturbs or subsamples attributes to show how sensitive the reported accuracy and interpretability are to attribute quality.","section":"Section 5, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the core idea is reasonable, but the evaluation currently mixes test-set tuning with the headline numbers and leaves the OT plan's hyperparameters unspecified. The main risk to acceptance is not the method's validity but the integrity of the empirical comparisons; I would like the authors to re-run or re-report with a clean tuning protocol and the missing controls before a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is nice: random multi-scale crops, LLM-generated attributes, and an OT plan to align the two sets, all without training. That specific combination is new relative to DCLIP and PatchCT, and the paper is clearly written, with code and ablations for each component. The accuracy improvements over DCLIP are real in the tables, and the domain generalization result on ImageNet-A is striking if it holds up.\n\nThe soft spots are mostly about evidence, not math. The Sinkhorn entropic regularization λ and the iteration count k are never reported. That matters because if λ is large the plan collapses toward uniform and Eq. 15 becomes a weighted average of crop-to-attribute similarities—indistinguishable from DCLIP with random-crop ensembling. The qualitative evidence in Fig. 3 is consistent with that worry: within each example, all attribute scores are nearly identical (≈0.067). The ablations only add components cumulatively; there is no control that replaces OT with plain averaging, so the causal role of local alignment is not established. Also missing: error bars despite stochastic cropping, and the hyperparameters α and θ are tuned on the same CUB and ImageNet benchmarks used for the final claims (Sec. 4.5). Minor provenance issues: Table 1 includes ProAPO without it being listed among baselines, and the CUB baseline in Table 3 (57.8) disagrees with DCLIP in Table 1 (57.1). These are fixable but should be cleaned up.\n\nThe central mechanism—OT as a way to localize attribute evidence—is plausible and worth testing, but the paper currently does not prove that the OT plan is doing anything more than averaging. That is a load-bearing omission, not a fatal flaw.\n\nThe paper deserves a serious referee. The right outcome is likely conditional acceptance after the authors report λ and k, add an averaging control, provide error bars, and re-tune hyperparameters on a validation split. I would bring it to a reading group, and I would cite it if the OT degeneracy gets resolved.","headline":"LaZSL is a plausible training-free recipe for interpretable ZSL, but the paper under-specifies the OT solver and never shows the transport plan is actually non-uniform, leaving open that the gains are just multi-crop ensembling.","tokens_in":14731,"tokens_out":2091,"would_cite":false,"duration_ms":24146,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that zero-shot classification improves and becomes interpretable when local image regions are aligned with LLM-generated attributes through an optimal transport plan, with no extra training.","keywords":["zero-shot learning","interpretability","vision-language models","optimal transport","local visual-semantic alignment","CLIP","large language models","domain generalization"],"falsifier":"A concrete test: replace the random-crop visual set with fixed grid patches, or with crops taken from a region known to contain the attribute, and re-run LaZSL. If accuracy does not drop when the crops are decorrelated from attribute locations, then the OT alignment is not actually using local evidence; alternatively, if randomly shuffling the rows of the cost matrix destroys the gain, the specific alignment is doing the work.","tokens_in":13766,"feed_emoji":"🎯","tokens_out":3729,"duration_ms":38067,"temperature":0.7,"pith_summary":"This paper proposes LaZSL, a training-free way to make vision-language models like CLIP both more accurate and more interpretable in zero-shot classification. Instead of comparing a whole image to class names or to global attribute descriptions, LaZSL crops the image into many regions, generates a set of visual attributes per class with a large language model, and uses optimal transport to match each region to the attributes it most plausibly supports. The match is combined with the global image-text similarity for the final class score. The paper shows on nine datasets that this locally aligned scoring beats prior interpretable baselines and improves robustness to domain shift. If correct, LaZSL implies that fine-grained visual-semantic correspondence, not just holistic similarity, is a practical source of zero-shot performance without additional training.","feed_headline":"Matching image regions to attributes beats global zero-shot scoring","feed_subtitle":"Training-free LaZSL uses optimal transport to align crops with LLM attributes, lifting accuracy and interpretability.","key_machinery":"The load-bearing mechanism is the optimal transport plan $T$ computed with the Sinkhorn algorithm between two discrete distributions: the visual set formed by random multi-scale crops of the query image and the semantic set formed by LLM-generated attributes of a candidate class. Three design choices carry the argument: a vision-selection step that zeroes out the probability mass of crops whose cosine similarity to the global image embedding falls below the average (Eqs. 6-9), a hybrid cost matrix that linearly blends region-attribute similarity with global image-attribute similarity (Eq. 10), and the Frobenius-inner-product scoring of the final plan against the same hybrid similarity (Eqs. 15-16). These components together let the model align fine-grained visual evidence with specific attributes while retaining the global context of the frozen CLIP encoder.","core_discovery":"The paper's central claim is that local visual-semantic alignment, realized as an optimal transport plan between a set of random multi-scale image crops and a set of LLM-generated attribute descriptions, yields better zero-shot classification than global image-text matching while also making predictions explainable. For each class, LaZSL builds a semantic set of attributes and a visual set of cropped regions, filters out regions that look unrelated to the global image, and solves a Sinkhorn-regularized optimal transport problem whose cost matrix mixes region-attribute similarity with global image-attribute similarity. The resulting transport plan provides both a per-region-to-attribute correspondence, which serves as the interpretable evidence, and a category score via a Frobenius inner product with the hybrid similarity. The paper reports consistent gains over DCLIP, CuPL, and WaffleCLIP across three CLIP backbones, and improved average accuracy on ImageNet domain-shift benchmarks, including a large gain on ImageNet-A.","pith_inferences":["The paper does not directly verify that the regions selected by the optimal transport plan actually contain the object parts named by the attributes; a dedicated attribution study comparing OT-selected regions with human-annotated part locations would test whether the interpretability is causal or merely correlational.","The Sinkhorn regularization and random-crop sampling introduce stochasticity; averaging over multiple crops or seeds could further stabilize accuracy and offers a low-cost variance-reduction extension.","The hybrid coefficient $\\theta = 0.8$ is fixed across all datasets; an adaptive $\\theta$ per dataset or per class might yield further gains, especially on fine-grained datasets where local information dominates.","The method's dependence on LLM attribute quality is acknowledged as a limitation; a filtering mechanism that scores attributes by their transport contribution could remove noisy descriptions and potentially improve both accuracy and interpretability."],"forward_implications":["LaZSL should allow CLIP-based zero-shot classifiers to show which attribute of a predicted class is supported by which image region, giving a per-prediction explanation without any training.","Because the method applies to any frozen VLM with patch-level visual features and an attribute generator, interpretability gains can transfer to newer backbones without retraining.","The domain-generalization improvement, especially on ImageNet-A, suggests that attribute-region alignment is more robust to natural distribution shift than global prompt similarity.","Since no training is involved, LaZSL can be deployed on new classes simply by generating attributes with an LLM, keeping the same cost profile as description-based baselines plus a small optimal-transport overhead."],"supporting_citations":[{"why":"Provides the LLM attribute-generation protocol and the interpretable baseline (DCLIP) that LaZSL extends and compares against.","marker":"[30]"},{"why":"Supplies the frozen CLIP encoders that produce the global, regional, and semantic features used throughout LaZSL.","marker":"[40]"},{"why":"Gives the Sinkhorn algorithm, the entropic-regularized optimal transport solver used to compute the alignment plan.","marker":"[13]"},{"why":"Introduces CuPL, an interpretable baseline that generates customized prompts via LLMs and serves as a comparison target.","marker":"[37]"},{"why":"Introduces WaffleCLIP, an interpretable baseline using random words and broad concepts, used for comparison.","marker":"[42]"},{"why":"Motivates the use of conditional transport for aligning patch sets with label sets in a vision-language context.","marker":"[28]"},{"why":"Supplies the optimal transport theory and computational background that the proposed alignment relies on.","marker":"[36]"}],"fun_headline_variants":["Local alignment makes zero-shot models explainable and better","Aligning image regions to attributes: no training, more accuracy","Optimal transport aligns crops to attributes for zero-shot","LaZSL: region-attribute matching boosts zero-shot and interpretability","Training-free local alignment sharpens zero-shot predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that random multi-scale crops, after being filtered by similarity to the whole image, contain the local visual evidence described by the LLM-generated attributes, so that the optimal transport plan reflects a true visual-semantic correspondence rather than noise.","fun_headline_variants_meta":{"raw":{"variants":["Local alignment makes zero-shot models explainable and better","Aligning image regions to attributes: no training, more accuracy","Optimal transport aligns crops to attributes for zero-shot","LaZSL: region-attribute matching boosts zero-shot and interpretability","Training-free local alignment sharpens zero-shot predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1843,"prompt_tokens":935,"completion_tokens":908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":827}},"tokens_in":551,"tokens_out":908,"duration_ms":8280,"temperature":1.0,"reasoning_tokens":827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:30:25.467598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: replace the random-crop visual set with fixed grid patches, or with crops taken from a region known to contain the attribute, and re-run LaZSL. If accuracy does not drop when the crops are decorrelated from attribute locations, then the OT alignment is not actually using local evidence; alternatively, if randomly shuffling the rows of the cost matrix destroys the gain, the specific alignment is doing the work.","supporting_citations":[{"cited_title":"Visual classification via description from large language models","cited_arxiv_id":null,"evidence_quote":"Provides the LLM attribute-generation protocol and the interpretable baseline (DCLIP) that LaZSL extends and compares against."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP encoders that produce the global, regional, and semantic features used throughout LaZSL."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Gives the Sinkhorn algorithm, the entropic-regularized optimal transport solver used to compute the alignment plan."},{"cited_title":"What does a platypus look like? generating customized prompts for zero- shot image classification","cited_arxiv_id":null,"evidence_quote":"Introduces CuPL, an interpretable baseline that generates customized prompts via LLMs and serves as a comparison target."},{"cited_title":"Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata","cited_arxiv_id":null,"evidence_quote":"Introduces WaffleCLIP, an interpretable baseline using random words and broad concepts, used for comparison."},{"cited_title":"Patchct: Align- ing patch set and label set with conditional transport for multi-label image classification","cited_arxiv_id":null,"evidence_quote":"Motivates the use of conditional transport for aligning patch sets with label sets in a vision-language context."},{"cited_title":"Computational optimal transport","cited_arxiv_id":null,"evidence_quote":"Supplies the optimal transport theory and computational background that the proposed alignment relies on."}],"review_version":1}