CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization

Dasol Hong; Hyun Myung; Wooju Lee

REVIEW 4 major objections 6 minor 13 references

CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that confidence-weighted prompt mixing and a confusion-aware loss let prompt-tuned CLIP improve specialization and generalization at the same time.

desk verdict A useful empirical prompt-tuning recipe with public code and solid results, but the advertised proof of a win-win mixture is a Jensen upper bound that does not deliver what the abstract claims. read the letter →

arxiv 2506.07484 v1 pith:2TO4FBZ7 submitted 2025-06-09 cs.CV cs.AIcs.LG

Dasol Hong , Wooju Lee , Hyun Myung This is my paper

classification cs.CVcs.AIcs.LG

keywords prompttuningvision-languagemodelsCLIPmixturemodelconfusion-awarelossconfidence-awareweightsfew-shotclass-incrementallearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that prompt tuning can become more specialized to trained classes and more general to unseen classes in the same update, rather than trading one off against the other. It proposes CoA-loss, which adds $1 - \hat{p}(y)$ to standard cross-entropy so that samples near confusing decision boundaries receive larger gradients. It then proves an upper bound showing that the expected error of a mixture of generalized and specialized prompts is no more than the weighted average of the prompts' individual errors, and argues that confidence-aware weights tighten that bound on specialization and generalization terms simultaneously. If the argument is right, prompt tuning no longer needs extra parameters or regularization to balance the two objectives; the reported experiments support this with average harmonic-mean gains of 15.28% in base-to-new generalization and 3.28% in cross-dataset transfer over zero-shot CLIP, plus a 5.6 percentage-point average accuracy gain in few-shot class-incremental learning.

What carries the argument

The carrying object is the mixture predictor $\hat{p}^{\pi}_{\mathcal{T}}$: a weighted combination of one generalized hand-crafted prompt $t_0$ and $K$ learned specialized prompts $t_i$, where each prompt contributes a class-similarity score and the weights $\pi$ decide how much each prompt is trusted before the scores are normalized into probabilities. Theorem 3.2 bounds the expected error of this mixture by the weighted average of the component prompts' errors, $\epsilon_{\mathcal{T}}(\hat{p}^{\pi}_{\mathcal{T}}) \le \sum_i \pi_i \epsilon_{\mathcal{T}}(\hat{p}_{t_i})$; Lemma 3.3 splits that bound into specialization terms, each prompt on its own sub-domain, and generalization terms, each prompt on other sub-domains. CoA-weights realize the weight assignment: $\pi^{\mathrm{in}}_i$ is tuned by cross-entropy over each training sub-domain, while $\pi^{\mathrm{out}}_i$ is tuned by an entropy-margin loss comparing specialized and generalized predictions on randomly sampled out-class words, making specialized predictions less confident outside their domain. CoA-loss is the specialization engine, $L = L_{\mathrm{CE}} + w(1 - \hat{p}(y))$, whose gradient grows for confusing samples and shrinks for clear ones.

What would settle it

Train CoCoA-Mix on a dataset where two learned sub-domain prompts cross-dominate, meaning each beats the other outside its own sub-domain, or where a specialized prompt outperforms the generalized prompt on unseen classes; then the confidence-aware weighting rule would lower the weight of the better predictor. A direct check is to run the method on a fine-grained target domain whose unseen classes are visually close to the training classes and compare new-class accuracy with zero-shot CLIP; if the assumption's ordering is violated often enough, the reported generalization gain should invert or disappear.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that specialization and generalization can be optimized together in one loss rather than traded off. The paper constructs a mixture model of one generalized prompt and several specialized prompts and proves that its expected error is bounded by a convex combination of the individual prompts' errors. It decomposes that bound into specialization error, where each prompt acts on its own sub-domain, and generalization error, where each prompt acts on other sub-domains, and shows that under Assumption 3.4, a specialized prompt being best in its own domain and the generalized prompt being best on unseen classes, increasing in-domain weights and decreasing out-domain weights lowers both error terms. CoA-loss implements the specialization side, while CoA-weights implement the generalization side. The empirical claim is that the combined method outperforms the compared prompt-tuning baselines across base-to-new generalization, cross-dataset transfer, and few-shot class-incremental learning.

Load-bearing premise

The whole win-win result rests on Assumption 3.4: a specialized prompt is always better than other prompts on its own sub-domain, and the generalized prompt is always better than specialized prompts on unseen classes; the paper validates this ordering only on CIFAR-100, not on the 11 datasets where the gains are claimed.

Editorial extensions

If this is right

Base-to-new generalization: on 11 datasets, the method raises the average harmonic mean of base and new accuracy by 15.28% relative to zero-shot CLIP, meaning specialization and generalization improve together rather than one at the other's expense.
Cross-dataset transfer: a prompt tuned on ImageNet transfers to 10 other datasets with an average harmonic-mean improvement of 3.28% over zero-shot CLIP, while keeping the highest source accuracy among compared methods.
Few-shot class-incremental learning: accumulating one specialized prompt per session with CoA-weights gives an average CIFAR-100 accuracy of 83.5%, 5.6 percentage points above zero-shot CLIP, with the lowest performance drop among the compared methods.
Parameter efficiency: the confidence-aware weighting uses two learnable scalars, whereas the compared sample-aware weight generator uses over 205,000 parameters, and the ablation attributes the generalization gain specifically to the entropy-margin optimization of the out-domain weight.
Loss design: CoA-loss outperforms cross-entropy, focal loss, generalized cross-entropy, and CE-plus-MAE on base classes by concentrating gradients on confusing samples, while the CoA-weights supply the additional gain on new classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper stops at textual prompts, but its own stated limitation, that the bound assumes consistent visual embeddings across styles, suggests a natural next test: adding visual-prompt optimization could extend the same confidence-weighting recipe from class shifts to style shifts.
The entropy-margin trick uses randomly sampled English words as stand-ins for unseen classes; if that proxy is good, gains should correlate with how semantically aligned a dataset's unseen classes are with generic vocabulary, a prediction the paper does not test.
The mixture error bound is generic, so any frozen classifier with a strong zero-shot predictor and a fine-tuned specialist could apply the same two-scalar confidence weighting, which points beyond CLIP to other embedding-based models.
The result implies a graded rather than binary outcome: on domains where Assumption 3.4 holds only weakly, the optimal mixture weight should move less aggressively toward the specialized prompt, a hypothesis that could be checked by measuring per-dataset margins between specialized and generalized predictions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

You should know about this paper for its empirical results, not for its theory. The recipe is simple: add (1 - p_correct) to cross-entropy with a weight w, and combine a specialized prompt with a generalized prompt using two scalar confidence weights that are tuned. The authors report strong base-to-new and FSCIL numbers, release code, and need only a handful of learnable parameters. That part is real and worth building on.

The theoretical claim is where I part ways. The paper says it "mathematically demonstrates" that the mixture error stays below the minimum error of individual prompts. Theorem 3.2 is a standard Jensen bound: the mixture error is at most the weighted average of the individual errors. A weighted average cannot be below the minimum of the terms unless the weights are negative. So the theorem cannot support the "lower than the minimum" conclusion. Lemma 3.3 just rewrites the same bound over sub-domains. The confidence-aware weight optimization is a reasonable heuristic that minimizes that upper bound, but it is not a proof that the mixture wins on both specialization and generalization. This is not a minor misstatement; it is the headline claim of the paper. The appendix's statistical check of Assumption 3.4 on CIFAR-100 is nice supporting evidence, but one dataset cannot carry the assumption across the eleven used in the main experiments.

The paper does several things well. The gradient analysis for CoA-loss is clear and the ablation separating pi_in and pi_out is informative. The loss-function comparison (focal, GCE, MAE, CE+MAE) is useful for the community. The FSCIL session-by-session table is honest, and the limitation section correctly flags the style-shift issue that explains the modest cross-dataset gains.

My net: the method is likely to be a useful baseline; the experiments are reproducible and the ablations are careful. But the theoretical framing is overstated and should be corrected in revision. I would referee this, and I would cite it after the claim is fixed. The assumption should either be validated on more domains or the claim downgraded to "empirically works across the benchmark."

Recommendation: send it out for review, but ask the authors to restate the theorem as a motivating bound and to soften the win-win language.

Referee Report

4 major / 6 minor

Summary. The paper proposes CoCoA-Mix, a prompt-tuning method for vision-language models that combines a confusion-aware loss (CoA-loss), defined as cross-entropy plus w(1-p_correct), with a two-prompt mixture model whose weights are made confidence-aware (CoA-weights). The in-domain weight is trained with cross-entropy on the mixture, while the out-domain weight is trained with an entropy margin loss computed over a set of randomly sampled words used as a proxy for unseen classes. The authors claim a mathematical guarantee that the mixture model improves generalization without sacrificing specialization, and report empirical gains over zero-shot CLIP on base-to-new generalization, cross-dataset transfer, and few-shot class-incremental learning.

Significance. If the empirical results hold, CoCoA-Mix is a lightweight and practical contribution: it adds no new network components, uses only scalar prompt-mixture weights, and is evaluated across three standard benchmarks with public code. The provided experiments and ablations are extensive and the main empirical comparisons are informative. However, the central theoretical claim is not established by the proof in the paper, and the gradient analysis in Eq. (11) contains an algebraic error. The value of the paper at this stage is therefore primarily empirical, not mathematical.

major comments (4)

[Abstract, Section 3.2, Theorem 3.2] The paper claims to 'mathematically demonstrate that a mixture model can enhance generalization without compromising specialization' and, in the contribution list, that the mixture error is 'lower than the minimum error of individual prompts.' Theorem 3.2 proves only the upper bound epsilon_T(p_hat^pi_T) <= sum_i pi_i epsilon_T(p_hat_t_i). Since a weighted average of errors is at least the minimum individual error, this inequality cannot imply epsilon_T(p_hat^pi_T) <= min_i epsilon_T(p_hat_t_i). Lemma 3.3 merely rewrites the same upper bound in terms of specialization and generalization errors, and Section 3.4 optimizes the weights to minimize that upper bound; minimizing an upper bound does not guarantee that the mixture beats every individual prompt. The mathematical 'win-win' claim is therefore unsupported by the stated theorem, even if Assumption 3.4 is valid.
[Section 3.3, Eq. (11)] The gradient with respect to the correct-class similarity s_t(y) is algebraically incorrect. From L_prompt = -log p(y) + w(1-p(y)), the derivative is dL/ds_t(y) = -(1/τ)(1-p(y))(1 + w p(y)), not -(1/τ)(1-p(y))(1 - w p(y)). For w > 0 and moderate p(y), the printed formula has the wrong sign: for example, with w=5 and p(y)=0.5 it gives +0.75/τ, under which gradient descent would decrease s_t(y) and lower the correct-class probability, opposite to the intended behavior. The incorrect-class gradient in Eq. (11) is correct. The plot in Figure 3 and the subsequent 'confusing cases receive larger gradients' argument are based on this erroneous expression and should be revised.
[Appendix B, Assumption 3.4] Assumption 3.4 is the premise that a specialized prompt is always better on its own sub-domain and that the generalized prompt is always better on unseen classes. This assumption underpins the design of the confidence-aware weights and the claimed absence of a trade-off. It is validated only on CIFAR-100 with 10 random splits, while the method is evaluated on 11 datasets with very different characteristics, including fine-grained and domain-shift datasets. The paper should validate the assumption on the same evaluation datasets or supply a theoretical condition under which the weight optimization is safe. As written, the 'no trade-off' conclusion is not supported for the reported benchmarks.
[Section 3.4, Eqs. (14)-(15), Appendix F.3] The out-class set used to optimize the out-domain weight pi_out is generated by sampling random words (Section 4.1), and the entropy loss treats this set as a proxy for unseen classes. No evidence is provided that the distribution of random words approximates the distribution of true unseen classes in the target domains, especially under the large domain shifts of cross-dataset transfer. The ablation in Table 9 only shows that Random Word beats Random String on base-to-new New accuracy; it does not establish robustness of the proxy. Because pi_out is the main mechanism targeting generalization, the paper should test the approximation more directly, for example by comparing random-word proxies against held-out real class names on several target datasets.

minor comments (6)

[Section 3.4 and Appendix B] The label 'Theorem 3.4' is used where 'Assumption 3.4' is meant; this appears both in the main text and in Appendix B and should be corrected.
[Figure 3] After correcting Eq. (11), the curves in Figure 3 should be regenerated, and the value of the temperature tau used in the plot should be stated.
[Abstract] The reported improvements of 15.28% and 3.28% are relative increases over zero-shot CLIP, not absolute percentage-point gains; this should be stated explicitly to avoid misreading.
[Section 4.2, Table 4] The term 'naive ensemble' is used before being defined; please state explicitly that it refers to uniform weighting of the individual prompts.
[Appendix F.3] The random-word generation relies on the Wonderwords API; for reproducibility, specify the package version and the random seed used for sampling.
[Section 4.1 and Appendix D.4] The FSCIL experiments use a ViT-L/14 backbone while base-to-new and cross-dataset experiments use ViT-B/16; this backbone difference is described only in the appendix and should be mentioned in the main experimental setup.

Circularity Check

1 steps flagged · score 4.0 of 10

No circularity in the mixture bound; the only construction-level issue is that CoA-loss is identical, up to weight rescaling, to the CE+MAE baseline the paper compares against, so one central component is a renamed known result.

renaming known result [Section 3.3, Eq. (9); Appendix E.3, Eq. (20)]
"LCoA(x, y; ˆpt) = 1 − ˆpt(y). The overall loss Lprompt for optimizing the prompt t is given by: Lprompt(x, y; ˆpt) = LCE + wLCoA. ... LCE+MAE(x, y; ˆp) = −log ˆp(y) + w 1/|Y| X l∈Y |1l=y − ˆp(l)|."

For one-hot labels, Σl |1l=y − p(l)| = (1−p(y)) + Σl≠y p(l) = 2(1−p(y)), so Eq. (20) is exactly CE + (2w/|Y|)(1−p(y)), i.e., the same function as Lprompt = CE + w(1−p(y)) up to a rescaling of the hyperparameter w. Thus the CoA-loss contribution (Eq. 9) is identical by construction to the CE+MAE baseline that Section E.3 treats as a separate, inferior loss. The claimed specialization improvement of CoA-loss over CE+MAE therefore reduces to re-labeling a known loss combination rather than introducing a new mechanism. The mixture-model analysis and CoA-weights experiments remain independent of this issue.

full rationale

The paper's mixture-model derivation is not circular: Theorem 3.2 proves a genuine Jensen upper bound, and Lemma 3.3 is just a re-grouping of that bound; neither is equivalent to the input by definition. The skeptical concern that the theorem does not imply the claimed error-below-best-prompt win-win is a logical overclaim about the bound, not a circularity, and I do not score it as one. There is no load-bearing self-citation: the external citations (Jensen 1906, Nguyen et al. 2022) are used as standard mathematical/domain-adaptation facts. The empirical results on held-out base/new splits, cross-dataset targets, and FSCIL sessions are genuine held-out evaluations, so the main accuracy claims are not fitted-input-called-prediction. The one construction-level problem is the renaming step: CoA-loss is algebraically identical to the CE+MAE loss the paper itself compares against, so this component is a renamed known result. That central specialization loss is a major claimed contribution, but the mixture and confidence-weighting contributions retain independent empirical content, giving a moderate overall circularity score of 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It uses a two-prompt mixture with scalar weights and a random-word pseudo-class set; the latter is an ad hoc construct whose validity is only partially tested.

free parameters (4)

w (CoA-loss weight) = 5.0 (base-to-new), 7.0 (cross-dataset)
Weight of CoA-loss in Lprompt = LCE + w LCoA, tuned per task (Appendix D.3, D.5).
d (margin in entropy loss) = 0.2 (base-to-new), 0.1 (FSCIL)
Margin in LEnt for entropy difference between specialized and generalized prompts over the out-class set (Eq. 15).
Cross-dataset pi_in loss weight = 2.0
Weight for the cross-entropy loss on pi_in in cross-dataset transfer (Appendix D.5).
Prompt length M = 16 (base-to-new), 2 (FSCIL)
Number of learnable tokens per prompt, a design choice per task (Sections 4.1 and D.4).

assumptions (4)

domain assumption Specialized prediction ti is no worse than other prompts on its own domain Di, and generalized t0 is no worse than specialized prompts on unseen classes.
Core to the claimed win-win of CoA-weights; empirically supported only on CIFAR-100 in Appendix B, not on the evaluation datasets. See Assumption 3.4.
standard math The domain adaptation bound from Nguyen et al. (2022): target error is bounded by source error plus a KL term plus a small conditional misalignment term.
Adopted as-is in Eq. (7) to justify minimizing source error; no proof is given in this paper.
domain assumption Visual embeddings for the same semantic class are similar across domains and styles.
Needed for source-to-target transfer; the authors acknowledge in Section H that CLIP may violate this, which weakens cross-dataset gains.
ad hoc to paper The random-word out-class set approximates the distribution of real unseen classes.
The entropy loss (Eq. 15) uses pseudo-out-classes from a random word API; its effectiveness is validated only indirectly via an ablation (Table 9), not against real unseen-class distributions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization." pith.science (2026). https://pith.science/paper/2TO4FBZ7

@misc{pith2026250607484,
  author       = {Pith},
  title        = {Pith review of: CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TO4FBZ7}},
  note         = {Machine review of arXiv:2506.07484}
}

read the original abstract

Prompt tuning, which adapts vision-language models by freezing model parameters and optimizing only the prompt, has proven effective for task-specific adaptations. The core challenge in prompt tuning is improving specialization for a specific task and generalization for unseen domains. However, frozen encoders often produce misaligned features, leading to confusion between classes and limiting specialization. To overcome this issue, we propose a confusion-aware loss (CoA-loss) that improves specialization by refining the decision boundaries between confusing classes. Additionally, we mathematically demonstrate that a mixture model can enhance generalization without compromising specialization. This is achieved using confidence-aware weights (CoA-weights), which adjust the weights of each prediction in the mixture model based on its confidence within the class domains. Extensive experiments show that CoCoA-Mix, a mixture model with CoA-loss and CoA-weights, outperforms state-of-the-art methods by enhancing specialization and generalization. Our code is publicly available at https://github.com/url-kaist/CoCoA-Mix.

Figures

Figures reproduced from arXiv: 2506.07484 by the authors.

**Figure 2.** CoCoA-Mix framework integrates confusion-aware loss (CoA-loss) for specialization and confidence-aware [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Gradient component of Lprompt with respect to (a) st(c ̸= y) and (b) st(y), where w = 0 represents standard cross-entropy. where w is a hyperparameter that balances the contribution of CoA-loss. To illustrate how CoA-loss refines the decision boundary between confusing classes, we analyze the gradients for the correct class y and an incorrect class c. The gradients of Lprompt with respect to the similarities st(y) a… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Comparison of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: (a) Proportion of predictions by zero-shot CLIP on EuroSAT. (b) Accuracy on easy test samples correctly predicted [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Box plots of accuracy differences across [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Performance improvement on confusing samples, defined as instances with a probability gap [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: Performance comparison across various loss functions with varying hyperparameters. The hyperparameter setting [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: Sensitivity analysis of hyperparameters. The dotted line [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

**Figure 10.** Figure 10: ScoreCAM results on Flowers102 and FGVCAircraft. Red indicates higher activation. The leftmost column [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

[1]

CALTECH101 100Object recognition Variety of object categories with random background images

π1 τBase NewH One-Stage Optimization τ0 τ1+τ0 τ1·τ0 τ1+τ0 79.375.1 77.0 Two-Stage Optimization exp(α1) exp(α0)+exp(α1) 0.0179.475.077.0 Table 6: 11 datasets used for base-to-new generalization and cross-dataset transfer Dataset CLASSESTASKDESCRIPTIONEXAMPLECLASSES IMAGENET 1,000Object recognition Large-scale dataset for object classifica- tion with divers...

work page 2000
[10]

In theFGVC Aircraftdataset (Base), CoA-loss focuses more precisely on fine-grained details such as text on airplane wings, outperforming zero-shot CLIP in specialization

In the Flowers102dataset (New), CoCoA-Mix more accurately attends to semantically meaningful regions in out-class samples, suggesting that CoA-weights effectively enhance generalization. In theFGVC Aircraftdataset (Base), CoA-loss focuses more precisely on fine-grained details such as text on airplane wings, outperforming zero-shot CLIP in specialization....

work page 2022
[11]

CoCoA-Mix used prompts of length M= 2 per session and accumulated them across sessions, requiring fewer parameters than the baseline except in the final two sessions

backbone. CoCoA-Mix used prompts of length M= 2 per session and accumulated them across sessions, requiring fewer parameters than the baseline except in the final two sessions. Each prompt was trained for specialization within its session, and the final prediction used all prompts from previous sessions. Tuned prompt ti for each session, along with CoA-we...

work page 2017
[12]

(2021), comparing CoCoA-Mix with zero-shot CLIP and CoOp

using the PyTorch library by Gildenblat et al. (2021), comparing CoCoA-Mix with zero-shot CLIP and CoOp. The visualization results are shown in Figure

work page 2021
[32]

CoA-weights were optimized using SGD with the same learning rate, a momentum of 0.9, and a weight decay of 5×10 −4

The prompt t was optimized using the Adam optimizer with a learning rate of 0.002 and a weight decay of 5×10 −4. CoA-weights were optimized using SGD with the same learning rate, a momentum of 0.9, and a weight decay of 5×10 −4. The weight for LCoA was set to w= 5.0 , the weight for LEnt was set to 10.0, and the margin was set tod= 0.2. The prompt lengthM...

work page 2024
[2009]

An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

arXiv 2010
[2014]

Unifying vision- and-language tasks via text generation

Cho, J., Lei, J., Tan, H., and Bansal, M. Unifying vision- and-language tasks via text generation. InInternational Conference on Machine Learning, pp. 1931–1942,

work page 1931
[2017]

UCF101: A dataset of 101 human ac- tions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

Soomro, K. UCF101: A dataset of 101 human ac- tions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

arXiv

Show all 13 references

[2020]

OFA: Unifying architec- tures, tasks, and modalities through a simple sequence- to-sequence learning framework

Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. OFA: Unifying architec- tures, tasks, and modalities through a simple sequence- to-sequence learning framework. InInternational Con- ference on Machine Learning, volume 162, pp. 233...

arXiv
[2021]

Open-vocabulary object detection via vision and language knowledge dis- tillation.arXiv preprint arXiv:2104.13921,

Gu, X., Lin, T.-Y ., Kuo, W., and Cui, Y . Open-vocabulary object detection via vision and language knowledge dis- tillation.arXiv preprint arXiv:2104.13921,

arXiv
[2022]

U., Rasheed, H., Maaz, M., Khan, S., and Khan, F

Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113–19122, 2023a. Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., a...

arXiv
[2023]

Exploring visual prompts for adapting large-scale models

Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274,

arXiv
[2024]

Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

CALTECH101 100Object recognition Variety of object categories with random background images

π1 τBase NewH One-Stage Optimization τ0 τ1+τ0 τ1·τ0 τ1+τ0 79.375.1 77.0 Two-Stage Optimization exp(α1) exp(α0)+exp(α1) 0.0179.475.077.0 Table 6: 11 datasets used for base-to-new generalization and cross-dataset transfer Dataset CLASSESTASKDESCRIPTIONEXAMPLECLASSES IMAGENET 1,000Object recognition Large-scale dataset for object classifica- tion with divers...

work page 2000

[2] [10]

In theFGVC Aircraftdataset (Base), CoA-loss focuses more precisely on fine-grained details such as text on airplane wings, outperforming zero-shot CLIP in specialization

In the Flowers102dataset (New), CoCoA-Mix more accurately attends to semantically meaningful regions in out-class samples, suggesting that CoA-weights effectively enhance generalization. In theFGVC Aircraftdataset (Base), CoA-loss focuses more precisely on fine-grained details such as text on airplane wings, outperforming zero-shot CLIP in specialization....

work page 2022

[3] [11]

CoCoA-Mix used prompts of length M= 2 per session and accumulated them across sessions, requiring fewer parameters than the baseline except in the final two sessions

backbone. CoCoA-Mix used prompts of length M= 2 per session and accumulated them across sessions, requiring fewer parameters than the baseline except in the final two sessions. Each prompt was trained for specialization within its session, and the final prediction used all prompts from previous sessions. Tuned prompt ti for each session, along with CoA-we...

work page 2017

[4] [12]

(2021), comparing CoCoA-Mix with zero-shot CLIP and CoOp

using the PyTorch library by Gildenblat et al. (2021), comparing CoCoA-Mix with zero-shot CLIP and CoOp. The visualization results are shown in Figure

work page 2021

[5] [32]

CoA-weights were optimized using SGD with the same learning rate, a momentum of 0.9, and a weight decay of 5×10 −4

The prompt t was optimized using the Adam optimizer with a learning rate of 0.002 and a weight decay of 5×10 −4. CoA-weights were optimized using SGD with the same learning rate, a momentum of 0.9, and a weight decay of 5×10 −4. The weight for LCoA was set to w= 5.0 , the weight for LEnt was set to 10.0, and the margin was set tod= 0.2. The prompt lengthM...

work page 2024

[6] [2009]

An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929,

arXiv 2010

[7] [2014]

Unifying vision- and-language tasks via text generation

Cho, J., Lei, J., Tan, H., and Bansal, M. Unifying vision- and-language tasks via text generation. InInternational Conference on Machine Learning, pp. 1931–1942,

work page 1931

[8] [2017]

UCF101: A dataset of 101 human ac- tions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

Soomro, K. UCF101: A dataset of 101 human ac- tions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

arXiv