REVIEW 4 cited by
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
read the original abstract
The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.
Forward citations
Cited by 4 Pith papers
-
Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Models
Three published hyperbolic vision-language models operate near-Euclidean with inoperative entailment cones and no detectable radial hierarchy, because the entailment objective admits a low-curvature shortcut.
-
All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs
LLM tasks are supported by multiple distinct circuits rather than unique mechanisms, demonstrated via Overlap-Aware Sheaf Repulsion and the Distributive Dense Circuit Hypothesis.
-
Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Models
Audited MERU, HyCoCLIP, and PHyCLIP checkpoints remain near-Euclidean with saturated cones and no operative shuffle-controlled radial hierarchy; entailment admits a low-curvature shortcut.
-
Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
HiR² extracts coarse-to-fine visual features from LMM layers and regularizes them with Lorentz entailment cones and unit-sphere dispersive loss, improving hierarchical consistency across models and fine-tuning methods.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.