pith. sign in

arxiv: 2607.01657 · v1 · pith:3ONDDIUPnew · submitted 2026-07-02 · 💻 cs.CV

Domain Generalization via Text-Anchored Information Bottleneck

Pith reviewed 2026-07-03 16:42 UTC · model grok-4.3

classification 💻 cs.CV
keywords domain generalizationinformation bottlenecklanguage embeddingsvision-language modelsinvariant representationsspurious cuesvisual recognitionrobustness
0
0 comments X

The pith

Language embeddings serve as the primary source of domain invariance by functioning as an information bottleneck that preserves semantics while suppressing environment-specific cues.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that visual expressiveness from vision-language models can propagate spurious cues that tie learned representations to training environments, which undermines invariant learning in domain generalization. By discarding visual guidance entirely and anchoring instead in language embedding space, the method creates a natural bottleneck that retains core task semantics and filters out domain-specific variations. This leads to stronger generalization across multiple backbones and datasets. A sympathetic reader would care because the result reframes domain generalization away from richer visual features and toward simpler supervision choices that enforce invariance.

Core claim

The central claim is that language embedding space acts as the primary source of domain invariance in visual recognition, naturally functioning as an information bottleneck that preserves core semantics while suppressing domain-specific variations; discarding visual guidance from vision-language models therefore yields state-of-the-art domain generalization performance.

What carries the argument

The text-anchored information bottleneck, in which language embeddings alone enforce invariance by serving as the main supervisory signal instead of visual features from vision-language models.

If this is right

  • Representations learned under language-only guidance exhibit reduced sensitivity to training-environment statistics.
  • Performance gains appear consistently across diverse backbone architectures when visual guidance is removed.
  • The focus of domain generalization shifts from enhancing visual representation capacity to designing supervision signals that enforce invariance.
  • Language embeddings suppress domain-specific variations more effectively than expressive visual features in the tested settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same language-anchoring principle could be tested in other multimodal settings where one modality introduces environment-specific correlations.
  • If language embeddings prove sufficient, future work might explore lighter text-only models for robustness tasks instead of full vision-language models.
  • The approach raises the question of whether other non-visual anchors, such as attribute labels or captions, could produce comparable bottlenecks.

Load-bearing premise

Language embeddings contain enough task-critical semantic information to support generalization even after visual expressiveness is removed and its associated spurious cues are eliminated.

What would settle it

A controlled experiment on a standard domain-generalization benchmark in which replacing visual guidance with language-embedding guidance produces lower accuracy on unseen target domains than the visual-guidance baseline.

Figures

Figures reproduced from arXiv: 2607.01657 by Eunyi Lyou, Joonseok Lee, Junho Lee, Yunjeong Choi.

Figure 1
Figure 1. Figure 1: (a) Visual encoders inevitably absorb spurious domain cues alongside domain￾invariant semantics. This inflates class regions and blurs boundaries, hindering robust￾ness. Ideally, models should drop domain-specific variations while preserving only core semantics. (b) Redefining invariance via a text-anchored Information Bottleneck (IB). Textual guidance acts as a semantic filter, preserving information shar… view at source ↗
Figure 2
Figure 2. Figure 2: Motivational experiments. (a) Textual embedding space (bottom) shows superior stability across domains than visual counterpart (top). (b) Visual encoders (gray) tend to possess both domain-specific and core-class information, while textual ones (blue) contain only the latter. (c) Text-guided models tend to yield lower Lipschitz constants, indicating smoother and more stable representations. black boxes), t… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Text-Anchored Information Bottleneck [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: DG performance across diverse backbones. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Information flow (X → Z → Y ) visualization using t-SNE on PACS. Left: For the house class, our method (blue) collapses inputs from different domains into a single cluster, while RISE (orange) retains domain-dependent separation, showing reduced cross-domain variation. Right: Within a single domain, our embeddings form more compact, well-separated class clusters, showing sharper class margins [PITH_FULL_I… view at source ↗
Figure 6
Figure 6. Figure 6: (a) I(Z; X|Y ) decreases with the regularizers Lalign, Lcomp during training. (b) Training with the regularizers achieves lower semantic loss and higher accuracy. (c) Accuracy decreases as the image guidance ratio increases. trained only with Lsem. By filtering domain-specific variations, the model allows Lsem to converge to a lower value, resulting in higher accuracy. Does visual guidance reintroduce doma… view at source ↗
read the original abstract

Visual recognition models often fail when deployed in new environments. Domain Generalization (DG) addresses this by learning representations that remain invariant to environment-specific variations. Recent approaches increasingly rely on large vision-language models, assuming that preserving their expressive visual representations improves robustness. However, we show that such visual expressiveness can instead propagate spurious cues that tie representations to the training environments, hindering invariant learning. We therefore discard visual guidance and instead treat the language embedding space as the primary source of domain invariance, naturally acting as an information bottleneck that preserves core semantics while suppressing domain-specific variations. Extensive experiments across diverse backbones exhibit state-of-the-art performance and further analyze what makes guidance effective for robust generalization. These findings shift the focus of DG from improving representations to designing supervision that enforces invariance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that expressive visual representations from vision-language models propagate spurious domain-specific cues that hinder invariant learning in domain generalization. It proposes discarding visual guidance in favor of language embedding space as the primary invariance source, which naturally functions as an information bottleneck to preserve task semantics while suppressing domain variations. The work reports state-of-the-art results across diverse backbones and analyzes factors that make such guidance effective.

Significance. If the central mechanism and empirical results hold, the paper would meaningfully shift DG research away from maximizing visual expressiveness toward supervision design that enforces invariance. This could influence how VLMs are leveraged in robustness settings and provide a new lens on information bottlenecks in multimodal DG.

major comments (2)
  1. [Abstract] Abstract: the claim that language embeddings 'naturally act as an information bottleneck' is presented without any loss formulation, objective, or derivation; without these it is impossible to determine whether the bottleneck property is emergent or engineered and whether it is load-bearing for the reported gains.
  2. [Abstract] The abstract asserts that visual expressiveness 'propagates spurious cues' and that discarding visual guidance yields SOTA performance, yet supplies neither the ablation isolating this causal path nor the quantitative comparison against visual-guidance baselines; these controls are required to substantiate the paradigm shift.
minor comments (1)
  1. The phrase 'text-anchored' in the title is not defined or operationalized in the provided text; a brief clarification of the anchoring mechanism would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful comments. We address each major point below, clarifying the manuscript content and proposing targeted revisions to the abstract where the presentation can be strengthened.

read point-by-point responses
  1. Referee: [Abstract] Abstract: the claim that language embeddings 'naturally act as an information bottleneck' is presented without any loss formulation, objective, or derivation; without these it is impossible to determine whether the bottleneck property is emergent or engineered and whether it is load-bearing for the reported gains.

    Authors: The abstract is a concise summary. Section 3 of the full manuscript provides the explicit loss formulation and objective: we anchor visual features to language embeddings via a contrastive objective that minimizes domain-specific mutual information while preserving task semantics, thereby engineering the bottleneck property rather than relying on emergence. This formulation is load-bearing, as shown by the ablation studies. We will revise the abstract to include a brief reference to the text-anchored objective. revision: yes

  2. Referee: [Abstract] The abstract asserts that visual expressiveness 'propagates spurious cues' and that discarding visual guidance yields SOTA performance, yet supplies neither the ablation isolating this causal path nor the quantitative comparison against visual-guidance baselines; these controls are required to substantiate the paradigm shift.

    Authors: The abstract summarizes the central claims. The manuscript substantiates them with ablations and quantitative comparisons in Section 4, including direct controls that isolate the effect of discarding visual guidance (versus retaining it) and demonstrate SOTA results across multiple backbones. To strengthen the abstract's standalone clarity, we will revise it to reference the empirical validation of the causal path and performance gains. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The abstract and provided context contain no equations, loss formulations, or derivation steps. The central claim—that language embeddings serve as a natural information bottleneck—is presented as a design choice motivated by empirical observation of visual spurious cues, not as a formally derived result that reduces to its own inputs or a self-citation chain. No self-definitional, fitted-prediction, or uniqueness-theorem patterns are identifiable. The paper is therefore self-contained against external benchmarks with no load-bearing circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that language embeddings inherently suppress domain-specific variations while retaining semantics; no free parameters or invented entities are mentioned in the abstract.

axioms (1)
  • domain assumption Language embedding space naturally acts as an information bottleneck that preserves core semantics while suppressing domain-specific variations.
    Directly stated in the abstract as the justification for discarding visual guidance.

pith-pipeline@v0.9.1-grok · 5661 in / 1253 out tokens · 26482 ms · 2026-07-03T16:42:58.547897+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 1 internal anchor

  1. [1]

    In: CVPR (2024)

    Addepalli, S., Asokan, A.R., Sharma, L., Babu, R.V.: Leveraging vision-language models for improving domain generalization in image classification. In: CVPR (2024)

  2. [2]

    In: NeurIPS (2021)

    Ahuja, K., Caballero, E., Zhang, D., Bengio, Y., Mitliagkas, I., Rish, I.: Invariance principle meets information bottleneck for out-of-distribution generalization. In: NeurIPS (2021)

  3. [3]

    In: ICLR (2017)

    Alemi, A.A., Fischer, I., Dillon, J.V., Murphy, K.: Deep variational information bottleneck. In: ICLR (2017)

  4. [4]

    Invariant Risk Minimization

    Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv:1907.02893 (2020)

  5. [5]

    In: NeurIPS (2022)

    Arpit, D., Wang, H., Zhou, Y., Xiong, C.: Ensemble of averages: Improving model selection and boosting performance in domain generalization. In: NeurIPS (2022)

  6. [6]

    In: ECCV (2018)

    Beery, S., Van Horn, G., Perona, P.: Recognition in terra incognita. In: ECCV (2018)

  7. [7]

    In: ICML (2018)

    Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., Hjelm, D.: Mutual information neural estimation. In: ICML (2018)

  8. [8]

    Machine Learning79(2010)

    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.: A theory of learning from different domains. Machine Learning79(2010)

  9. [9]

    In: NIPS (2006)

    Ben-David, S., Blitzer, J., Crammer, K., Pereira, F.: Analysis of representations for domain adaptation. In: NIPS (2006)

  10. [10]

    In: NeurIPS (2021)

    Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y., Park, S.: SWAD: Domain generalization by seeking flat minima. In: NeurIPS (2021)

  11. [11]

    In: ECCV (2022)

    Cha, J., Lee, K., Park, S., Chun, S.: Domain generalization by mutual-information regularization with pre-trained models. In: ECCV (2022)

  12. [12]

    In: ECCV (2020)

    Chattopadhyay, P., Balaji, Y., Hoffman, J.: Learning to balance specificity and invariance for in and out of domain generalization. In: ECCV (2020)

  13. [13]

    CVPR (2024)

    Cheng, D., Xu, Z., Jiang, X., Wang, N., Li, D., Gao, X.: Disentangled prompt representation for domain generalization. CVPR (2024)

  14. [14]

    In: ICLR (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)

  15. [15]

    In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M

    Du, Y., Xu, J., Xiong, H., Qiu, Q., Zhen, X., Snoek, C.G.M., Shao, L.: Learn- ing to learn with variational information bottleneck for domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV (2020)

  16. [16]

    MIT Press (1968)

    Fano, R.M.: Transmission of Information: A Statistical Theory of Communication. MIT Press (1968)

  17. [17]

    Entropy22(9), 999 (Sep 2020) Domain Generalization via Text-Anchored Information Bottleneck 17

    Fischer, I.: The conditional entropy bottleneck. Entropy22(9), 999 (Sep 2020) Domain Generalization via Text-Anchored Information Bottleneck 17

  18. [18]

    Proceedings of the royal society of London

    Fisher, R.A.: Dispersion on a sphere. Proceedings of the royal society of London. Series A. Mathematical and physical sciences217(1130), 295–305 (1953)

  19. [19]

    In: ICML

    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: ICML. pp. 1180–1189 (2015)

  20. [20]

    JMLR (2016)

    Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. JMLR (2016)

  21. [21]

    In: ICLR (2021)

    Gulrajani, I., Lopez-Paz, D.: In search of lost domain generalization. In: ICLR (2021)

  22. [22]

    In: ICCV (2023)

    Guo, J., Qi, L., Shi, Y.: DomainDrop: Suppressing domain-sensitive channels for domain generalization. In: ICCV (2023)

  23. [23]

    In: CVPR (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

  24. [24]

    ICCV (2023)

    Huang, Z., Zhou, A., Lin, Z., Cai, M., Wang, H., Lee, Y.J.: A sentence speaks a thousand images: Domain generalization through distilling CLIP with language guidance. ICCV (2023)

  25. [25]

    CVPR (2023)

    Jain, S., Addepalli, S., Sahu, P.K., Dey, P., Babu, R.V.: DART: Diversify- aggregate-repeat training improves generalization of neural networks. CVPR (2023)

  26. [26]

    In: ICCV (2023)

    Jeon,M.,Kang,M.,Lee,J.:Aunifiedframeworkforrobustnessondiversesampling errors. In: ICCV (2023)

  27. [27]

    In: CVPR (2022)

    Jeon, M., Kim, D., Lee, W., Kang, M., Lee, J.: A conservative approach for unbi- ased learning on unknown biases. In: CVPR (2022)

  28. [28]

    arXiv:2210.03117 (2022)

    khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: MaPLe: Multi-modal prompt learning. arXiv:2210.03117 (2022)

  29. [29]

    ICCVW (2023)

    Lew, B., Son, D., Chang, B.: Gradient estimation for unseen domain risk mini- mization with pre-trained models. ICCVW (2023)

  30. [30]

    In: AAAI (2021)

    Li, B., Shen, Y., Wang, Y., Zhu, W., Reed, C., Zhang, J., Li, D., Keutzer, K., Zhao, H.: Invariant information bottleneck for domain generalization. In: AAAI (2021)

  31. [31]

    In: ICCV (2017)

    Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: ICCV (2017)

  32. [32]

    In: CVPR (2018)

    Li, H., Pan, S.J., Wang, S., Kot, A.C.: Domain generalization with adversarial feature learning. In: CVPR (2018)

  33. [33]

    arXiv preprint arXiv:2203.04600 (2022) 5

    Li, Z., Ren, K., Jiang, X., Li, B., Zhang, H., Li, D.: Domain generalization using pretrained models without fine-tuning. arXiv:2203.04600 (2022)

  34. [34]

    arXiv:2308.09931 (2023)

    Liu, G.M., Wang, Y.: TDG: Text-guided domain generalization. arXiv:2308.09931 (2023)

  35. [35]

    In: CVPR (2022)

    Lv, F., Liang, J., Li, S., Zang, B., Liu, C.H., Wang, Z., Liu, D.: Causality inspired representation learning for domain generalization. In: CVPR (2022)

  36. [36]

    IJCV (Dec 2023)

    Mao, X., Chen, Y., Jia, X., Zhang, R., Xue, H., Li, Z.: Context-aware robust fine- tuning. IJCV (Dec 2023)

  37. [37]

    In: ICML (2013)

    Muandet, K., Balduzzi, D., Schölkopf, B.: Domain generalization via invariant feature representation. In: ICML (2013)

  38. [38]

    ICLR (2024)

    Nam, G.C., Heo, B., Lee, J.: Lipsum-FT: Robust fine-tuning of zero-shot models using random text guidance. ICLR (2024)

  39. [39]

    Lyou et al

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Syn- naeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual fe...

  40. [40]

    In: ICCV (2019)

    Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: ICCV (2019)

  41. [41]

    Neurocomputing (2021)

    Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A.W., Luong, M.T., Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning. Neurocomputing (2021)

  42. [42]

    In: ICML (2020)

    Piratla, V., Netrapalli, P., Sarawagi, S.: Efficient domain generalization via common-specific low-rank decomposition. In: ICML (2020)

  43. [43]

    The MIT Press (2009)

    Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press (2009)

  44. [44]

    In: ICML (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)

  45. [45]

    In: ICML (2023)

    Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., Long, M.: CLIPood: Generalizing CLIP to out-of-distributions. In: ICML (2023)

  46. [46]

    In: NIPS (2020)

    Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: MPNet: masked and permuted pre- training for language understanding. In: NIPS (2020)

  47. [47]

    In: ECCV (2016)

    Sun, B., Saenko, K.: Deep CORAL: Correlation alignment for deep domain adap- tation. In: ECCV (2016)

  48. [48]

    of the Annual Allerton Conference on Communication, Control and Computing (1999)

    Tishby,N., Pereira, F.C., Bialek, W.: Theinformation bottleneckmethod.In: Proc. of the Annual Allerton Conference on Communication, Control and Computing (1999)

  49. [49]

    In: CVPR (2011)

    Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR (2011)

  50. [50]

    Wiley-Interscience (1998)

    Vapnik, V.N.: Statistical Learning Theory. Wiley-Interscience (1998)

  51. [51]

    In: CVPR (2017)

    Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: CVPR (2017)

  52. [52]

    CVPR (2023)

    Wang, P., Zhang, Z., Lei, Z., Zhang, L.: Sharpness-aware gradient matching for domain generalization. CVPR (2023)

  53. [53]

    In: NeurIPS (2020)

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MINILM: deep self- attention distillation for task-agnostic compression of pre-trained transformers. In: NeurIPS (2020)

  54. [54]

    arXiv:2507.12851 (2025)

    Wang, Z., Gao, Z., Chen, J., Zhao, Q., Wu, X., Luo, J.: Simulate, refo- cus and ensemble: An attention-refocusing scheme for domain generalization. arXiv:2507.12851 (2025)

  55. [55]

    In: CVPR (2025)

    Wen, C., Peng, Z., Huang, Y., Yang, X., Shen, W.: Domain generalization in CLIP via learning with diverse text prompts. In: CVPR (2025)

  56. [56]

    In: CVPR (2022)

    Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., Schmidt, L.: Robust fine-tuning of zero-shot models. In: CVPR (2022)

  57. [57]

    In: CVPR (2025)

    Yang, D., Lee, J., Kim, Y.: TAROT: Towards essentially domain-invariant robust- ness with theoretical justification. In: CVPR (2025)

  58. [58]

    CVPR (2023)

    Yu, H., Zhang, X., Xu, R., Liu, J., He, Y., Cui, P.: Rethinking the evaluation protocol of domain generalization. CVPR (2023)

  59. [59]

    IEEE TIP (2023)

    Yu, X., Tseng, H.H., Yoo, S., Ling, H., Lin, Y.: INSURE: An information theory inspired disentanglement and purification model for domain generalization. IEEE TIP (2023)

  60. [60]

    In: NeurIPS (2024)

    Yu, X., Yoo, S., Lin, Y.: CLIPCEIL: Domain generalization through CLIP via channel refinement and image-text alignment. In: NeurIPS (2024)

  61. [61]

    Transactions of the Japanese Society for Artificial Intelligence (2021)

    Zhang, X., Iwasawa, Y., Matsuo, Y., Gu, S.S.: Domain prompt learning for effi- ciently adapting CLIP to unseen domains. Transactions of the Japanese Society for Artificial Intelligence (2021)

  62. [62]

    CVPR (2022) Domain Generalization via Text-Anchored Information Bottleneck 19

    Zhang, X., Zhou, L., Xu, R., Cui, P., Shen, Z., Liu, H.: NICO++: Towards better benchmarking for domain generalization. CVPR (2022) Domain Generalization via Text-Anchored Information Bottleneck 19

  63. [63]

    In: CVPR (2022)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: CVPR (2022)

  64. [64]

    USER:<image>Generate a short and clear caption for the image. ASSISTANT:

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV130(9), 2337–2348 (Jul 2022) Domain Generalization via Text-Anchored Information Bottleneck i Appendix A Details on Caption Generation To better understand the structure of CLIP text embeddings, we visualize image- conditioned captions generated usingLLaVA-v1.5-7B1 ...