Pith. sign in

REVIEW 4 major objections 5 minor 76 references

A text prompt can re-aim image similarity at any chosen attribute — color, style, camera angle — without touching the stored gallery embeddings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A single hypernetwork turns text descriptions of attributes into affine maps of frozen CLIP embeddings, making those attributes control retrieval and clustering without re-encoding the gallery.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Practical text-conditioned affine re-mapping of frozen CLIP embeddings that mostly delivers; missing error bars and a text-vs-lookup baseline, but the core idea is solid. the 4 major comments →

arxiv 2607.22919 v1 pith:MNGI4ESB submitted 2026-07-24 cs.CV cs.AI

Controlling Embedding Spaces with Text-Conditioned Transformations

Joseph Fioresi , Fabian Caba Heilbron , Pankaj Nathani , Mubarak Shah , Kushal Kafle This is my paper
classification cs.CV cs.AI
keywords embedding space controlattribute-aware retrievaltext-conditioned hypernetworkaffine transformationmulti-clusteringCLIPdisentanglementfrozen gallery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the attribute a user cares about in an image search can be selected at query time by reshaping the embedding space, with no retraining of the underlying model and no re-encoding of the gallery. The mechanism is a text-conditioned hypernetwork that reads a phrase such as "color" or "art style" and outputs a matrix-vector shift; applying that affine map to a query embedding makes the named attribute dominate similarity. The same map applied to an entire embedding set reorganizes it for attribute-specific clustering. A sympathetic reader would care because it promises intuitive, near-instant control over retrieval and organization from frozen CLIP-style features, and the paper demonstrates it across many attribute categories and multiple datasets.

Core claim

The paper's central claim is that attributes suppressed by CLIP's dominant-semantics compression remain linearly recoverable, and that a single learned affine transformation conditioned on text can amplify any chosen attribute axis. Concretely, the hypernetwork H_phi(t) outputs (W,b), and the transformed query is v' = vW^T + b; it is trained with a multi-positive contrastive loss to sit close to base embeddings sharing the attribute label, plus a multimodal alignment loss toward attribute text. At inference the same operator has two uses: applied to a query it yields attribute-aware retrieval against an unchanged index, and applied to the whole set it yields a new geometry in which k-means s

What carries the argument

The load-bearing object is the text-conditioned hypernetwork H_phi producing a global affine transformation (W,b) in R^{DxD} x R^D, applied as v' = vW^T + b. A three-layer MLP takes the text embedding of an attribute category and predicts W and b. Training uses a multi-positive contrastive loss that pulls each transformed query near base-embedding images sharing the attribute label, plus a symmetric text-alignment loss. Because the query-gallery score factors as v'^T W^T v + b^T v, the transformation can be folded into the query at scoring time while the stored gallery stays frozen. Singular-value analysis of W shows it concentrates similarity in a compact, attribute-specific subspace.

Load-bearing premise

Every attribute this method can emphasize is assumed to be recoverable by one global affine map of the frozen embedding space; if an attribute's signal is only nonlinearly recoverable, no single (W,b) can bring it out.

What would settle it

On a dataset with an attribute whose subclasses are linearly inseparable in CLIP features but separable by a small nonlinear probe, train the hypernetwork on that attribute and check whether query mAP improves over base features; if the affine transform fails where the nonlinear probe succeeds, the central premise is falsified. A quicker screen: compute the stable rank of learned W and test whether its top singular subspace aligns with linear separability of the attribute across held-out subclasses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Attribute-aware retrieval from a frozen gallery: users can type "camera angle" or "color" and get ranked results emphasizing that axis, with no re-encoding or storage overhead.
  • Attribute-specific clustering of the same collection: one learned transform, applied globally, gives k-means a geometry that separates the requested attribute, replacing slow caption-and-LLM pipelines.
  • Constant memory across attributes: one hypernetwork serves all trained attribute categories, so storage does not grow with the number of attributes the way per-attribute adapters do.
  • Compositional control: linearly blending per-attribute transformed queries yields joint attribute retrieval without retraining.
  • Transfer: transforms learned on synthetic or narrow domains improve attribute retrieval on real-world images, and the same mechanism works on text embeddings for sentiment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the affine-recoverability premise holds broadly, the top singular directions of W give a cheap, interpretable description of how any attribute is expressed in the frozen space; one could use them to probe what CLIP suppresses.
  • The authors note expressiveness is bounded by categories seen in training; a natural extension is meta-training on many attribute categories to test whether unseen attribute names produce useful transforms from text conditioning alone — an experiment they leave to future work.
  • Because the transform acts on embeddings rather than pixels, the same interface could extend to audio, video, or other latent spaces; the paper's text-sentiment result is one proof of concept pointing that way.
  • The linear-blend composition result hints at a vector-space algebra of attributes; if that algebra is consistent, one might edit a query by subtracting one attribute transform and adding another, e.g., changing style while keeping content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a text-conditioned hypernetwork H_phi(t) that generates an affine transformation (W,b), applied to frozen CLIP visual embeddings as v' = vW^T + b. In query mode, only the query embedding is transformed, enabling attribute-aware retrieval from an unchanged gallery; in global mode, the same transformation is applied to the full embedding set for attribute-specific multi-clustering. Training uses a multi-positive contrastive loss (Eq. 3) plus a multimodal alignment loss (Eq. 4), with a single hypernetwork covering 19 attributes and 187 subclasses across datasets. Experiments report large gains over existing baselines in attribute-aware retrieval (Table 1) and multi-clustering (Table 2), with ablations on transformation type, multimodal weight, data efficiency, attribute scalability, and backbone choice.

Significance. If the results hold, the method is a simple, efficient, and unified framework for attribute-level control of frozen embedding spaces: it requires no re-encoding of the gallery, no per-attribute fine-tuning, and supports many attributes through one network. The paper ships extensive experiments — per-category retrieval and clustering results in the supplement, generalization to unseen color classes, cross-dataset transfer to MS-COCO, and compatibility with a frozen backbone (DINOv2) — which strengthen the empirical case. The main claims are plausible and the training objective is clearly defined. However, the evidence is not yet fully rigorous: no error bars, no ablation isolating the text-conditioning contribution, and no test of the affine-family assumption. These gaps are addressable and should be fixed before publication.

major comments (4)
  1. [§4.4, Tables 1–2] No error bars or multiple-seed statistics are reported for any experimental result. The central state-of-the-art claims in Tables 1 and 2 rest on single runs. Given the large reported margins, the conclusions are plausible, but without variance estimates the comparison is not statistically grounded. Please report mean ± std over at least three seeds for the main tables and ablations.
  2. [§4.2, Table 1] There is no baseline that isolates the text-conditioning contribution. The hypernetwork is trained with a supervised contrastive loss on attribute labels; the text embedding is only the conditioning input. A baseline with a fixed or random text-conditioning vector, using the same hypernetwork and loss, would show whether the gains come from the learned affine projection or from text conditioning itself. The ‘−multimodal’ ablation in Table 8 removes Eq. (4) but does not remove text conditioning from Eq. (1).
  3. [§4.6, Table 3] The affine transformation family is a central design choice, and the paper’s justification — “Attributes are entangled in CLIP space, yet they remain linearly separable [4]” — is not sufficient. The cited SPLICE result concerns sparse linear directions for binary concept presence, not mutual separability of all K subclasses by a single affine metric. The paper never evaluates a nonlinear transformation. Please add an experiment with a nonlinear map (e.g., an MLP with the same training loss) to bound the effect of the affine assumption. If a nonlinear transform materially improves results, the claim of ‘controlling embedding spaces’ needs qualification.
  4. [§5, Conclusion] The conclusion states that “expressiveness is currently bounded by the range of categories seen during training.” This is an important limitation, but it is discussed only at the end. It should appear earlier and be expanded: the paper cannot claim fully general attribute control for attributes never seen at training time, only for attributes within the trained taxonomy. This limitation is distinct from the affine-family concern in §4.6 and should be separated clearly.
minor comments (5)
  1. [Fig. 1 caption] The caption contains an internal author note: “Kushal’s v2. Tried to put some more text idk if we need it.” Remove this before submission.
  2. [§3.2, Eq. 3 discussion] The score decomposition is notationally inconsistent with Eq. (2). With v′_i = v_i W^T + b, the score is v_i^T W^T v_j + b^T v_j. Rewrite the sentence after Eq. (3) to avoid the typo ⟨W v′_i + b, v⟩.
  3. [§4.3] Specify the hidden dimensions of the three-layer MLP hypernetwork, and state explicitly whether v′ is ℓ2-normalized before the contrastive losses and before retrieval similarity computation. Eq. (2) does not by itself produce a unit vector.
  4. [Supplementary, Table S6] The memory-footprint arithmetic is inconsistent: the text states an HNSW overhead factor of 2×, but the calculation uses 3GB × 1.5 ≈ 6GB. Also, 3GB × 1.5 = 4.5GB, not 6GB. Please correct.
  5. [§4.6] The sentence “Attributes are entangled in CLIP space, yet they remain linearly separable [4]” is a strong claim. Please quote or summarize the specific SPLICE result that supports it, since the reference is about binary concepts, not multi-class attribute subclasses.

Circularity Check

0 steps flagged

No significant circularity: the transform is supervised on labeled attribute subclasses and evaluated on held-out splits; no prediction reduces to a fitted constant.

full rationale

The derivation chain is self-contained. The hypernetwork H_phi is trained with a multi-positive contrastive loss (Eq. 3) and a multimodal alignment loss (Eq. 4) on labeled attribute subclasses, and the reported retrieval and clustering numbers are measured on held-out test splits, on a frozen gallery, and—in Table 7—on subclasses held out during training. The query-mode score is a linear re-weighting of the frozen-gallery inner product, and the global mode applies the same learned transform to all embeddings before unsupervised k-means; neither step reintroduces evaluation labels as fitted constants. The only external premise, linear separability of attributes in CLIP space, is cited to SPLICE [4], an independent prior work, and functions as an empirical assumption rather than as a result imported from the authors. Self-citations in the related-work section (e.g., [12,47,49,50]) are not load-bearing for the method's correctness. No equation reduces to its input by construction, and no fitted parameter is renamed as a prediction. The method is therefore not circular; its limitations (e.g., expressiveness bounded by the affine family and by training categories) are scope limitations, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The method is empirical and supervised; the ledger tracks the hand-set hyperparameters and the domain assumptions that the affine-sufficiency and text-conditioning claims depend on. The learned MLP weights themselves are not itemized because they are the trained model rather than ad hoc constants.

free parameters (6)
  • Contrastive temperature tau = not reported
    Appears in Eq. (3); a standard hyperparameter but its value is never given.
  • Multimodal loss weight omega = 0.1
    Selected by hand; Supp. Table S10 shows retrieval/clustering trade-off as omega varies.
  • Compositional blend weight alpha = 0.5 (Table 4), 0.8 (Supp Eq. 8)
    Hand-set for query blending and for the CIR reranking score.
  • Text template list = unspecified
    Eq. (4) averages text embeddings over templates such as "a photo of {c}"; the full template list is not given.
  • k-means cluster count k = true number of classes per attribute (e.g., 10 for Clevr4)
    Clustering accuracy is computed with Hungarian matching, which requires k equal to the ground-truth class count; the paper does not discuss unsupervised k selection.
  • Hypernetwork hidden dimensions = unspecified
    A 3-layer MLP is specified but hidden widths are not; capacity of the transform generator is a free choice.
axioms (4)
  • domain assumption Attributes are linearly separable in frozen CLIP space
    Stated in Section 4.6 citing SPLICE [4]; this is what justifies a single affine map.
  • domain assumption A single global affine map per attribute is sufficient for the whole distribution
    Eq. (2) applies the same W,b to every image; no per-image adaptation, and no nonlinear transform is tested (Table 3 only compares affine vs. translation).
  • domain assumption Text embedding of the attribute category is a sufficient conditioning signal
    The hypernetwork maps t = ft(T) to W,b; only held-out subclasses are tested, not unseen attribute categories, so text may act as a lookup key.
  • domain assumption Query-side training transfers to global clustering
    The global mode is called "emergent" and is never optimized; k-means on transformed embeddings must still find attribute structure.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlling Embedding Spaces with Text-Conditioned Transformations." pith.science (2026). https://pith.science/paper/MNGI4ESB

@misc{pith2026260722919,
  author       = {Pith},
  title        = {Pith review of: Controlling Embedding Spaces with Text-Conditioned Transformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNGI4ESB}},
  note         = {Machine review of arXiv:2607.22919}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., "color" or "art style"), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Project page: https://joefioresi718.github.io/ControlEmbed_webpage/

Figures

Figures reproduced from arXiv: 2607.22919 by Fabian Caba Heilbron, Joseph Fioresi, Kushal Kafle, Mubarak Shah, Pankaj Nathani.

Figure 1
Figure 1. Figure 1: Overview of our text-conditioned generation of embedding-space transforma￾tions. (Left) We propose a hypernetwork that maps natural-language attribute descrip￾tions to a modification function. Applying this transformation to a query embedding produces an attribute-aware representation that emphasizes the requested property. (Right) By controlling the embedding space with attribute transforms, the same mech… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our multi-attribute training workflow. An attribute category is sampled (e.g., “action” or “location”) and passed through the proposed hypernetwork to generate a transformation function, which is applied to a query embedding to pro￾duce attribute-focused representations aligned with base embeddings of images sharing the same category attribute label via a multi-positive contrastive loss. An add… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative attribute-aware retrieval results under different attribute trans￾formation functions. Our text-conditioned query transformations selectively retrieve images that match the specified attribute without losing the scene semantics at first, then prioritize the attribute over general semantics. From top to bottom, we show re￾trievals for "action", "count", then fine-grained perceptual property "cam… view at source ↗
Figure 5
Figure 5. Figure 5: Mean category mAP across datasets (each line) as the number of jointly trained attribute categories increases from a single category per model to unified training across all datasets. Performance remains consistent even as more attributes are added, demon￾strating that a single hypernetwork can scale across 14+ attributes without degradation. CLIP-B/16 CLIP-L/14 SigLIP-G/16 DINOv2-L/14 Qwen3-VL￾Embed-2B 0 … view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on backbone architecture, evaluated using retrieval mAP. Our method is effective across all types/sizes of encoders, including non-text aligned DINOv2. Backbone Encoder Choice [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: t-SNE visualization of embeddings colored by shot-size labels before and after transformation. The baseline space (left) shows entangled clusters. After applying our learned transformation (right), samples align intuitively along a continuous gradient that reflects the shot-size taxonomy. Can the transformation capture perceptual attributes and how does it affect embedding geometry? Contrastive image–text … view at source ↗
Figure 8
Figure 8. Figure 8: Our attribute-conditioned similarity metric helps characterize unique relation￾ships that CLIP similarity alone cannot. (Left) Baseline CLIP similarity prioritizes overall semantics. (Middle, Right) Applying attribute-specific transforms to the query (e.g., color or location) selectively re-scores relationships based on the input attribute. jects with uniform backgrounds, the model still produces meaningfu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 1 canonical work pages

  1. [1]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Agnolucci, L., Baldrati, A., Del Bimbo, A., Bertini, M.: isearle: Improving textual inversion for zero-shot composed image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  2. [2]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Baldrati, A., Agnolucci, L., Bertini, M., Del Bimbo, A.: Zero-shot composed image retrieval with textual inversion. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 15338–15347 (2023)

    2023

  3. [3]

    Baldrati, A., Bertini, M., Uricchio, T., Del Bimbo, A.: Conditioned and composed imageretrievalcombiningandpartiallyfine-tuningclip-basedfeatures.In:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4959–4968 (2022)

    2022

  4. [4]

    Advances in Neural Informa- tion Processing Systems37, 84298–84328 (2024)

    Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F.P., Lakkaraju, H.: Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Informa- tion Processing Systems37, 84298–84328 (2024)

    2024

  5. [5]

    In: European conference on computer vision

    Cao, B., Araujo, A., Sim, J.: Unifying deep local and global features for image search. In: European conference on computer vision. pp. 726–743. Springer (2020)

    2020

  6. [6]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z.D., Zhao, L.J., Zhang, Z.C., Luo, X., Xu, X.S.: Characteristics matching based hash codes generation for efficient fine-grained image retrieval. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17273–17281 (2024)

    2024

  7. [7]

    In: European Conference on Computer Vision

    Cui, Q., Jiang, Q.Y., Wei, X.S., Li, W.J., Yoshie, O.: Exchnet: A unified hashing network for large-scale fine-grained image retrieval. In: European Conference on Computer Vision. pp. 189–205. Springer (2020)

    2020

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019)

    2019

  9. [9]

    arXiv preprint arXiv:2401.08281 (2024)

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. arXiv preprint arXiv:2401.08281 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    International Journal of Computer Vision132(2), 581–595 (2024)

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., Qiao, Y.: Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision132(2), 581–595 (2024)

    2024

  11. [11]

    In: International conference on neural information processing

    Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., et al.: Challenges in representation learning: A report on three machine learning contests. In: International conference on neural information processing. pp. 117–124. Springer (2013)

    2013

  12. [12]

    Gupta, A., Parmar, J., Dave, I.R., Shah, M.: From play to replay: Composed video retrievalfortemporallyfine-grainedvideos.arXivpreprintarXiv:2506.05274(2025)

    arXiv 2025

  13. [13]

    In: Proceedings of the IEEE/CVF International conference on computer vision

    Hou, Y., Vig, E., Donoser, M., Bazzani, L.: Learning attribute-driven disentangled representations for interactive fashion retrieval. In: Proceedings of the IEEE/CVF International conference on computer vision. pp. 12147–12157 (2021)

    2021

  14. [14]

    ICLR1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)

    2022

  15. [15]

    Knowledge and Information Systems51(3), 991–1021 (2017)

    Hu, J., Qian, Q., Pei, J., Jin, R., Zhu, S.: Finding multiple stable clusterings. Knowledge and Information Systems51(3), 991–1021 (2017)

    2017

  16. [16]

    In: International conference on machine learning

    Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021) Controlling Embedding Spaces with Text-Conditioned Transformations 17

    2021

  17. [17]

    In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi-modal prompt learning. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  18. [18]

    Advances in neural information processing systems33, 18661–18673 (2020)

    Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems33, 18661–18673 (2020)

    2020

  19. [19]

    arXiv preprint arXiv:1412.6980 (2014)

    Kingma, D.P.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    Pith/arXiv arXiv 2014

  20. [20]

    Naval research logistics quarterly2(1-2), 83–97 (1955)

    Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2), 83–97 (1955)

    1955

  21. [21]

    Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., et al.: Matryoshka representa- tionlearning.AdvancesinNeuralInformationProcessingSystems35,30233–30249 (2022)

    2022

  22. [22]

    arXiv preprint arXiv:2310.18297 (2023)

    Kwon, S., Park, J., Kim, M., Cho, J., Ryu, E.K., Lee, K.: Image clustering condi- tioned on text criteria. arXiv preprint arXiv:2310.18297 (2023)

    Pith/arXiv arXiv 2023

  23. [23]

    Ad- vances in Neural Information Processing Systems35, 17612–17625 (2022)

    Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems35, 17612–17625 (2022)

    2022

  24. [24]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

    2014

  25. [25]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  26. [26]

    Liu, H., He, J., Jin, Y., Zheng, D., Dong, Y., Zhang, F., Huang, Z., He, Y., Li, Y., Chen, W., Qiao, Y., Ouyang, W., Zhao, S., Liu, Z.: Shotbench: Expert-level cinematic understanding in vision-language models (2025),https://arxiv.org/ abs/2506.21356, accessed: 21 July 2025

    arXiv 2025

  27. [27]

    arXiv preprint arXiv:2410.05217 (2025)

    Liu, M., Zhong, Z., Li, J., Franchi, G., Roy, S., Ricci, E.: Organizing unstructured image collections using natural language. arXiv preprint arXiv:2410.05217 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Liu, Z., Rodriguez-Opazo, C., Teney, D., Gould, S.: Image retrieval on real- life images with pre-trained vision-and-language models. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2125–2134 (2021)

    2021

  29. [29]

    arXiv preprint arXiv:2305.16304 (2023)

    Liu, Z., Sun, W., Teney, D., Gould, S.: Candidate set re-ranking for composed image retrieval with dual multi-modal encoder. arXiv preprint arXiv:2305.16304 (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  31. [31]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Lu,X.,Chen,S.,Cao,Y.,Zhou,X.,Lu,X.:Attributesgroupingandmininghashing for fine-grained image retrieval. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 6558–6566 (2023)

    2023

  32. [32]

    In: European Con- ference on Computer Vision

    Luo, Y., An, R., Zou, B., Tang, Y., Liu, J., Zhang, S.: Llm as dataset analyst: Subpopulation structure discovery with large language model. In: European Con- ference on Computer Vision. pp. 235–252. Springer (2025)

    2025

  33. [33]

    IEEE transactions on pattern analysis and machine intelligence42(4), 824–836 (2018)

    Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neigh- bor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence42(4), 824–836 (2018)

    2018

  34. [34]

    arXiv preprint arXiv:2210.07183 (2022) 18 J

    Menon, S., Vondrick, C.: Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 (2022) 18 J. Fioresi et al

    Pith/arXiv arXiv 2022

  35. [35]

    In: Proceedings of the AAAI conference on artificial intelligence

    Miklautz, L., Mautz, D., Altinigneli, M.C., Böhm, C., Plant, C.: Deep embedded non-redundant clustering. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 5174–5181 (2020)

    2020

  36. [36]

    arXiv preprint arXiv:1301.3781 (2013)

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word repre- sentations in vector space. arXiv preprint arXiv:1301.3781 (2013)

    Pith/arXiv arXiv 2013

  37. [37]

    In: Proceedings of the IEEE international conference on computer vision

    Noh, H., Araujo, A., Sim, J., Weyand, T., Han, B.: Large-scale image retrieval with attentive deep local features. In: Proceedings of the IEEE international conference on computer vision. pp. 3456–3465 (2017)

    2017

  38. [38]

    Scientific Data12(1), 254 (2025).https://doi.org/10.1038/s41597-025-04529-4,https://doi.org/ 10.1038/s41597-025-04529-4

    Ohm, T., Karjus, A., Tamm, M.V., Schich, M.: fruit-salad: A style aligned artwork dataset to reveal similarity perception in image embeddings. Scientific Data12(1), 254 (2025).https://doi.org/10.1038/s41597-025-04529-4,https://doi.org/ 10.1038/s41597-025-04529-4

    work page doi:10.1038/s41597-025-04529-4 2025

  39. [39]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    Pith/arXiv arXiv 2023

  40. [40]

    Advances in Neural Information Process- ing Systems32(2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chillemi, G., De- Pasquale, L., Gitman, T., Li, J., Lin, Z., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Process- ing Systems32(2019)

    2019

  41. [41]

    In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)

    Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word repre- sentation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014)

    2014

  42. [42]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  43. [43]

    IEEE Transactions on Knowledge and Data Engineering35(9), 8852–8864 (2022)

    Ren, L., Yu, G., Wang, J., Liu, L., Domeniconi, C., Zhang, X.: A diversified atten- tion model for interpretable multiple clusterings. IEEE Transactions on Knowledge and Data Engineering35(9), 8852–8864 (2022)

    2022

  44. [44]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Saini, N., Pham, K., Shrivastava, A.: Disentangling visual embeddings for at- tributes and objects. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13658–13667 (2022)

    2022

  45. [45]

    In: European conference on computer vision

    Shen, Y., Sun, X., Wei, X.S., Jiang, Q.Y., Yang, J.: Semicon: A learning-to-hash solution for large-scale fine-grained image retrieval. In: European conference on computer vision. pp. 531–548. Springer (2022)

    2022

  46. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Siméoni, O., Avrithis, Y., Chum, O.: Local features and visual words emerge in activations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11651–11660 (2019)

    2019

  47. [47]

    In: Computer Vision – ECCV 2024

    Sirnam, S., Yang, J., Neiman, T., Rizve, M.N., Tran, S., Yao, B., Chilimbi, T., Shah, M.: X-former: Unifying contrastive and reconstruction learning for mllms. In: Computer Vision – ECCV 2024. pp. 146–162. Springer Nature Switzerland (2025)

    2024

  48. [48]

    In: Proceedings of the 2013 conference on empirical methods in natural language processing

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1631–1642 (2013)

    2013

  49. [49]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2026) Controlling Embedding Spaces with Text-Conditioned Transformations 19

    Swetha, S., Meng, R., Ram, S., Neiman, T., Tran, S., Shah, M.: Smpro: Self- supervised visual preference alignment via differentiable multi-preference multi- group ranking. In: Proceedings of the AAAI Conference on Artificial Intelligence (2026) Controlling Embedding Spaces with Text-Conditioned Transformations 19

    2026

  50. [50]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Swetha, S., Rizve, M.N., Shvetsova, N., Kuehne, H., Shah, M.: Preserving modal- ity structure improves multi-modal learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 21993–22003 (October 2023)

    2023

  51. [51]

    Advances in Neural Information Processing Systems36, 48382–48402 (2023)

    Tian, Y., Fan, L., Isola, P., Chang, H., Krishnan, D.: Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems36, 48382–48402 (2023)

    2023

  52. [52]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

    Tian, Y., Newsam, S., Boakye, K.: Fashion image retrieval with text feedback by additive attention compositional learning. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1011–1021 (2023)

    2023

  53. [53]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9568–9578 (2024)

    2024

  54. [54]

    arXiv preprint arXiv:2502.14786 (2025)

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025)

    Pith/arXiv arXiv 2025

  55. [55]

    In: SIGGRAPH Asia 2023 Conference Papers

    Valevski, D., Lumen, D., Matias, Y., Leviathan, Y.: Face0: Instantaneously con- ditioning a text-to-image model on a face. In: SIGGRAPH Asia 2023 Conference Papers. pp. 1–10 (2023)

    2023

  56. [56]

    Advances in Neural Information Processing Systems 37 (2023)

    Vaze, S., Vedaldi, A., Zisserman, A.: No representation rules them all in category discovery. Advances in Neural Information Processing Systems 37 (2023)

    2023

  57. [57]

    In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition

    Veit, A., Belongie, S., Karaletsos, T.: Conditional similarity networks. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 830–838 (2017)

    2017

  58. [58]

    arXiv preprint arXiv:2507.14137 (2025)

    Venkataramanan, S., Pariza, V., Salehi, M., Knobel, L., Gidaris, S., Ramzi, E., Bursuc, A., Asano, Y.M.: Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint arXiv:2507.14137 (2025)

    Pith/arXiv arXiv 2025

  59. [59]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Vo, N., Jiang, L., Sun, C., Murphy, K., Li, L.J., Fei-Fei, L., Hays, J.: Composing text and image for image retrieval-an empirical odyssey. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6439–6448 (2019)

    2019

  60. [60]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, J., Xu, Y., Hu, J., Yan, M., Sang, J., Qian, Q.: Improved visual fine-tuning with natural language supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11899–11909 (2023)

    2023

  61. [61]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, L., Ao, W., Boddeti, V.N., Lim, S.N.: Generative zero-shot composed im- age retrieval. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29690–29700 (2025)

    2025

  62. [62]

    arXiv preprint arXiv:2509.17049 (2025)

    Wang, P., Li, Y., Zhao, L., Wei, X.S.: Learning attribute-aware hash codes for fine- grained image retrieval via query optimization. arXiv preprint arXiv:2509.17049 (2025)

    arXiv 2025

  63. [63]

    arXiv preprint arXiv:2602.00813 (2026)

    Wang, T., Zhao, Y., Kong, S.: Generating a paracosm for training-free zero-shot composed image retrieval. arXiv preprint arXiv:2602.00813 (2026)

    Pith/arXiv arXiv 2026

  64. [64]

    In: Proceedings of the AAAI conference on artificial intelligence

    Wei, S., Wang, J., Yu, G., Domeniconi, C., Zhang, X.: Multi-view multiple cluster- ings using deep matrix factorization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 6348–6355 (2020)

    2020

  65. [65]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(11), 13904–13920 (2023) 20 J

    Wei, X.S., Shen, Y., Sun, X., Wang, P., Peng, Y.: Attribute-aware deep hashing with self-consistency for large-scale fine-grained image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence45(11), 13904–13920 (2023) 20 J. Fioresi et al

    2023

  66. [66]

    Advances in Neural Infor- mation Processing Systems34, 5720–5730 (2021)

    Wei, X.S., Shen, Y., Sun, X., Ye, H.J., Yang, J.: A2-net: Learning attribute-aware hash codes for large-scale fine-grained image retrieval. Advances in Neural Infor- mation Processing Systems34, 5720–5730 (2021)

    2021

  67. [67]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., Zuo, W.: Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15943–15953 (2023)

    2023

  68. [68]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.: Robust fine-tuning of zero- shot models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7959–7971 (2022)

    2022

  69. [69]

    In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition

    Wu, H., Gao, Y., Guo, X., Al-Halah, Z., Rennie, S., Grauman, K., Feris, R.: Fash- ion iq: A new dataset towards retrieving images by natural language feedback. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 11307–11317 (2021)

    2021

  70. [70]

    In: 2011 International conference on computer vision

    Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L.: Human action recognition by learning bases of action attributes and parts. In: 2011 International conference on computer vision. pp. 1331–1338. IEEE (2011)

    2011

  71. [71]

    Procedia Computer Science222, 571–580 (2023)

    Yao, J., Liu, E., Rashid, M., Hu, J.: Augdmc: Data augmentation guided deep multiple clustering. Procedia Computer Science222, 571–580 (2023)

    2023

  72. [72]

    Advances in Neural Information Processing Systems37, 82705– 82725 (2024)

    Yao, J., Qian, Q., Hu, J.: Customized multiple clustering via multi-modal subspace proxy learning. Advances in Neural Information Processing Systems37, 82705– 82725 (2024)

    2024

  73. [73]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yao, J., Qian, Q., Hu, J.: Multi-modal proxy learning towards personalized visual multiple clustering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14066–14075 (2024)

    2024

  74. [74]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  75. [75]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, Z., Wang, L., Zhou, L., Koniusz, P.: Learning spatial-context-aware global visual feature representation for instance image retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11250–11259 (2023)

    2023

  76. [76]

    low angle & dutch angle

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16816–16825 (2022) Controlling Embedding Spaces with Text-Conditioned Transformations 21 Supplementary Overview Section A: Dataset details Section B: Implementation deta...

    2022

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.