SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

Haoji Zhang; Junsi Li; Ming Sun; Xinyu Mao; Yu Liang

REVIEW 4 major objections 5 minor 13 references

A patch-slimming framework guided by dense text from a multimodal language model sets a new state-of-the-art for fine-grained image-text retrieval.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:20 UTC pith:FH5SOBGW

load-bearing objection The method is a plausible next step, but the headline rSum gains are point differences mislabeled as percentages, and the ablation table contradicts itself; the numbers as reported cannot be trusted. the 4 major comments →

arxiv 2511.01390 v2 pith:FH5SOBGW submitted 2025-11-03 cs.CV cs.AIcs.MM

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

Xinyu Mao , Junsi Li , Haoji Zhang , Yu Liang , Ming Sun This is my paper

classification cs.CV cs.AIcs.MM

keywords image-text retrievalfine-grained cross-modal alignmentpatch selectiondense text supervisionmultimodal LLMtext-to-image retrievalsemantic relevancepatch slimming

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SEPS claims that the core obstacle to fine-grained image-text alignment is the information-density gap: images contain many redundant and ambiguous patches while captions are sparse. To bridge this gap, the framework generates a dense description of each image with a multimodal language model, then uses both the dense text and the original sparse caption to score, select, and aggregate the most relevant visual patches. A second module aligns those selected patches to words by combining mean pooling with a relevance-aware top-k reward, preventing irrelevant patches from diluting the similarity. On Flickr30K and MS-COCO, SEPS reports state-of-the-art retrieval, with rSum gains of 23%-86% over prior methods across different vision encoders and particularly large gains in text-to-image retrieval. If correct, this suggests that MLLM-generated dense text can serve as a cheap, offline preprocessing step to improve fine-grained retrieval without fine-tuning the language model.

Core claim

The core discovery is that dense text produced by a multimodal language model, when fused with the original sparse caption through a two-stage scoring and decision mechanism, can identify which visual patches are semantically relevant, thereby eliminating redundancy and resolving ambiguity. The SDTPS module computes per-patch significance from dense-text and sparse-text cross-attention plus image self-attention, applies Gumbel-Softmax to make differentiable selection decisions, and aggregates the surviving patches into a compact set. The HRPA module then computes patch-word similarities, takes max scores row- and column-wise, and adds a learned top-k reward to the mean of alignment scores, r

What carries the argument

The central machinery is the two-module pipeline: (1) SDTPS, which fuses sparse- and dense-text cross-attention scores with image self-attention into a single significance score, uses Gumbel-Softmax to create differentiable binary patch masks, and aggregates the selected patches via learned weight matrices; (2) HRPA, which computes a patch-word similarity matrix, applies max-pooling in both directions, and adds a learnable transformation of the top-k maxima to the mean of the remaining scores. The dense text is generated offline by an MLLM as a preprocessing step, so no gradients flow through the generator. The mechanism's role is to turn patch-word similarity from a global average into a se

Load-bearing premise

The framework assumes that the dense descriptions produced by a multimodal language model for each image are faithful, non-conflicting expansions of the original captions, and that using them at test time does not leak information about the benchmark's ground-truth captions.

What would settle it

Rerun the same framework but replace the dense text with (a) the original caption repeated verbatim or (b) random fluent text; if the retrieval gains vanish or reverse, the improvement depends on the MLLM's specific image-to-text knowledge rather than on text length alone. Additionally, check whether the MLLM was pretrained on images or captions from the test set; if so, the reported test-set numbers could be inflated by data leakage.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If correct, MLLM-generated dense descriptions can be used as a zero-training preprocessing step to improve fine-grained image-text retrieval without modifying the language model.
The reported gains imply that patch selection guided by dense text is more impactful for text-to-image retrieval than for image-to-text, suggesting that the bottleneck for T2I is visual patch ambiguity rather than caption sparsity.
The framework's ability to work across different vision encoder architectures and with a widely-used contrastive pretrained model suggests it could be a plug-in for existing retrieval pipelines.
The large gains on the larger MS-COCO 5K set indicate that dense-text guidance matters most in complex scenes, where patch redundancy is higher.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable extension: replace the MLLM-generated dense text with human-written detailed captions for the same images; if the gains persist, the benefit comes from text density rather than from the MLLM's learned image knowledge.
The relevance-aware selection mechanism could transfer to video-text alignment, where frame-level patches have even greater redundancy; one would predict the gains in text-to-video retrieval to be at least as large as those reported here.
Because the MLLM is kept frozen and queried with a single fixed prompt, the framework may be sensitive to the prompt's style; varying the prompt would reveal whether the improvement is tied to a particular type of description.
An ablation that varies the top-k size in the HRPA module could reveal a scaling law: the optimal k might track the average number of salient objects per image in the dataset.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The method is a plausible next step, but the headline rSum gains are point differences mislabeled as percentages, and the ablation table contradicts itself; the numbers as reported cannot be trusted.

read the letter

Quick take: the headline numbers do not survive contact with Table 1. The abstract/conclusion claim of 23–86% rSum improvement is actually a range of absolute point gains against various baselines, not relative percentages. Against the strongest prior method in each setting, relative rSum gains are roughly 4–19%. Section 4.4 then compounds the problem: the "13.8%" and "27.2%" R@1 improvements are point differences, and the "71.3% rSum improvement" is 71.3 points, which is about 16% relative. This is not a rounding issue; the central quantitative claim is a different quantity from what was measured.

What is good: the core idea—using MLLM-generated dense text to guide visual patch selection at the input level, rather than only for feature distillation—is a natural and sensible extension of LAPS and D2S-VSE. The SDTPS two-stage selection and HRPA relevance-aware alignment are clearly described and plausible. The CLIP-backbone extension in the appendix suggests the idea transfers. The authors promise code and checkpoints, which is the right thing to do.

The soft spots are serious. The ablation table has internal contradictions: "introduce aggregation" shows a large drop in R@1 compared to "without aggregation" (85.2→74.4 image-to-text, 84.7→62.7 text-to-image), while the text says aggregation gives a 2.2% gain. The "only dense text" row has image-to-text R@1 equal to R@5 (80.3/80.3), which is suspect. The default-configuration numbers in appendix Table 4 disagree with Table 1 for ViT-Base-224 (rSum 564.3 vs 560.9). Hyperparameters β, N_c, and TOPK in Equations 3–5 are never reported, despite the reproducibility statement. No error bars or multiple seeds are given.

There is also an open leakage question: using LLaVA to generate dense text for test images gives SEPS access to an external model that may have seen Flickr30K/COCO images, and the "no gradients" defense does not address that. The comparison to prior methods that do not use this extra test-time input is not apples-to-apples without a discussion, and that discussion is absent.

Bottom line: the underlying idea deserves exploration, but the current manuscript is not reliable as written. A revised version with honest percentage reporting, a corrected ablation table, standard deviations, and a real leakage analysis could be a solid paper. For now, treat the headline results as unverified.

Recommendation: send it to peer review—the idea is legitimate and there is enough substance to warrant referee time, but reviewers should be explicitly asked to check the numbers against Table 1 and the ablation table. My own verdict is reject in current form.

Referee Report

4 major / 5 minor

Summary. The paper proposes SEPS, a fine-grained image-text retrieval framework. It uses LLaVA to generate dense textual descriptions, then combines these with the original sparse captions in a two-stage patch-selection module (SDTPS), and finally computes image-text similarity with a Highly-Relevant Patch-Word Alignment (HRPA) module that combines max-based relevance selection with mean pooling. Experiments on Flickr30K and MS-COCO are reported across ViT and Swin backbones, and on a CLIP backbone in an appendix. The abstract and conclusion claim that SEPS surpasses existing methods by 23%–86% in rSum. I found the core idea plausible and the experimental scope broad, but the paper's headline quantitative claims are not supported by the data in Table 1, and several reproducibility and leakage issues remain unresolved.

Significance. If the results hold, the central idea — using MLLM-generated dense text to guide visual patch selection rather than only to enrich text features — is a timely and credible contribution to fine-grained cross-modal alignment. The paper includes useful experimental breadth: four visual encoders, two standard benchmarks, an extension to CLIP, ablations, and sensitivity analyses for ρ and λ. The authors also provide code and checkpoints. However, the main quantitative claim is miscalibrated, and the leakage and omitted-hyperparameter issues prevent the reader from assessing the true contribution. The paper is not ready in its current form.

major comments (4)

[Abstract; §4.4; Table 1] The headline claim of 23%–86% rSum improvement is not supported by Table 1. Relative to the strongest prior per setting (D2S-VSE), rSum gains are about 5.5% (Flickr30K 1K, ViT-Base-224), 6.7% (COCO 1K), 16.0% (COCO 5K), 6.3%/7.5%/19.0% for ViT-Base-384, and similar or lower for Swin variants. No setting reaches even 20% relative improvement. The '13.8%', '27.2%', and '71.3%' figures in §4.4 are absolute percentage-point differences (73.9−60.1, 73.5−46.3, 516.9−445.6), not relative improvements. Please correct the abstract and conclusion to state relative percentages, or explicitly describe these as absolute rSum point gains.
[§3.1; §4.3] The claim that 'no information leakage occurs from the test datasets' because dense-text generation has 'no gradients backward' does not address the relevant leakage path. LLaVA is pretrained on large-scale web data that may include MS-COCO images; using it to generate dense text for COCO test images can encode test-set content into the patch-selection signal. Freezing the generator does not prevent this. A control experiment is needed, e.g. using an MLLM with no COCO exposure, or evaluating on images that were definitely not in LLaVA's pretraining, or at minimum an explicit discussion and risk assessment.
[§3.2–3.3; §4.3] Key hyperparameters are not reported: β in Eq. (3), N_c in Eq. (4), and the TOPK size in Eq. (5) are absent from Section 4.3 and the appendix. Figure 3 and Tables 4–5 only report ρ, λ1, and λ2. Without values or a concrete selection rule for β, N_c, and TOPK, the method is not reproducible.
[§4.5, Table 2] The ablation table conflicts with Table 1. For Flickr30K with ViT-Base-224, Table 1 reports SEPS image-to-text R@5 = 93.7, while Table 2's 'completeSEPS' row reports 96.7. The 'only dense text' row gives image-to-text R@1 = R@5 = 80.3, which is impossible since recall@5 must be at least recall@1. Additionally, the text's '17.5% improvement' in Text-to-Image R@1 is not derivable from Table 2 (86.9 vs 67.2 is 19.7 points, or 29.3% relative). These inconsistencies undermine the ablation claims.

minor comments (5)

[Abstract; Conclusion] Mirroring Major Comment 1, the '23%-86%' phrase should be corrected after the quantitative claims are re-expressed.
[General] There are stray '/uni0000...' token sequences immediately after Table 1 and in the body text; these appear to be artifacts and should be removed.
[Eq. (5)] TOPK is used without definition. Please define it in the text or in a notation table, and give the value used in the experiments.
[Ethics Statement] The statement that LLMs were used to polish 'Section 4 and appendix' is vague. Specify which models were used and the extent of the polishing, if the venue requires such disclosure.
[Figure 3 caption] The caption mentions selection ratio ρ and coefficients λ1 and λ2, but not β or N_c. If these are fixed, state their values in the caption or nearby text.

Circularity Check

0 steps flagged

No circularity: SEPS is a benchmark-evaluated empirical pipeline; the SOTA claim is a measured result, not a definitional or self-citational reduction.

full rationale

SEPS does not present a formal derivation whose conclusion is equivalent to an input. Each component is either learned from data or borrowed from cited external work: LLaVA generates dense text (Sec. 3.1); SDTPS computes patch scores from learned predictions and attention similarities (Eqs. 1-3); the Gumbel-Softmax decision and aggregation follow LAPS and prior sampling methodology (Sec. 3.2.2); HRPA defines the image-text score S(I,T) via row/column maxima and MLP transforms (Eq. 5); training uses the standard hard-negative triplet loss plus a ratio constraint (Eqs. 6-7). The reported rSum values in Table 1 are measured evaluation metrics on held-out splits, not quantities forced by definitions or by fitted parameters renamed as predictions. The 'score-aware prediction network' predicts cross-attention scores for an internal auxiliary objective; it is not a benchmark prediction fitted to test data. The authors have no load-bearing self-citations, and no uniqueness theorem or ansatz is imported from their own prior work. The Sec. 4.3 'no information leakage' claim is a leakage/fairness argument about LLaVA's pretraining rather than a definitional cycle, and the Sec. 4.4 '23%-86%' statement is a numerical-reporting discrepancy relative to Table 1, which is a correctness concern rather than circularity. Under the rule that circularity requires exhibiting a specific reduction, none is present.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

SEPS is an empirical framework with several hand-set hyperparameters (β, λ1, λ2, ρ, N_c, TOPK, α) and strong domain assumptions about the reliability of MLLM dense text. It introduces no new theoretical entities. The most important unverified premise is the leakage-free use of LLaVA at test time.

free parameters (7)

β
Eq. 3 balances the learned per-patch score against three cross-attention scores; its value or learning rule is not reported.
λ1 = 1 (default), 0.5 (Table 5 optimum)
Sparse-text ratio-loss coefficient in Eq. 7; ablated in Table 5 and not derived.
λ2 = 1 (default), 0.5 (Table 5 optimum)
Dense-text ratio-loss coefficient in Eq. 7; ablated in Table 5 and not derived.
ρ = 0.5 (ViT), 0.8 (Swin)
Target patch-selection ratio in Eq. 7; ablated in Table 4 and chosen by hand.
N_c
Number of aggregated patches in Eq. 4; only constrained as less than max(N_s, N_d), no exact value given.
TOPK k
Dimension used for the TOPK features fed to the MLP in Eq. 5; not specified.
α = 0.2
Triplet margin in Eq. 6; set by hand following prior work.

axioms (4)

standard math Gumbel-Softmax provides a differentiable approximation to binary patch decisions
Invoked in Section 3.2.2 to justify differentiable decision matrices.
domain assumption Dense text generated by LLaVA is a faithful, conflict-free expansion of the sparse caption for the purpose of patch selection
The whole SDTPS module assumes the dense text encodes semantic information that can guide visual patch selection without contradicting the sparse text; introduced in Sections 3.1 and 3.2.
domain assumption Using LLaVA at test time does not leak benchmark supervision because no gradients are back-propagated
Section 4.3 states 'no information leakage occurs from the test datasets' solely on gradient grounds, but LLaVA is a pretrained external model whose training data may overlap with the benchmarks.
domain assumption Text-relevant patch selection is sufficient for fine-grained cross-modal alignment
The framework discards patches not deemed relevant by the dense/sparse text scoring, assuming the discarded patches do not carry necessary alignment signal; this underpins the SDTPS module.

pith-pipeline@v1.3.0-alltime-deepseek · 18847 in / 12722 out tokens · 135980 ms · 2026-08-04T00:20:28.464559+00:00 · methodology

0 comments

read the original abstract

Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications. Current approaches face challenges in addressing patch redundancy and ambiguity, which arise from the inherent information density disparities across modalities. Recently, Multimodal Large Language Models (MLLMs) have emerged as promising solutions to bridge this gap through their robust semantic generation capabilities. However, the dense textual outputs from MLLMs may introduce conflicts with the original sparse captions. Furthermore, accurately quantifying semantic relevance between rich visual patches and concise textual descriptions remains a core challenge. To overcome these limitations, we introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity. Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches. Additionally, it leverages relevance-aware selection with mean value computation to highlight crucial patch-word correspondences, thereby improving cross-modal similarity assessment. Comprehensive experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance, surpassing existing approaches by 23\%-86\% in rSum across diverse model architectures, with notable enhancements in text-to-image retrieval scenarios. Our implementation is available at https://github.com/Sweet4tars/seps.git.

Figures

Figures reproduced from arXiv: 2511.01390 by Haoji Zhang, Junsi Li, Ming Sun, Xinyu Mao, Yu Liang.

**Figure 2.** Figure 2: (a) Overview of our Semantic-Enhanced Patch Slimming(SEPS) Framework for fine [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: The retrieval performance of different selection ratios [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: The visualization of visual patch selection with different combinations of sparse text and [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

**Figure 5.** Figure 5: The visualization of cross-modal alignment results of SEPS. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 6 linked inside Pith

[1]

Uniter: Learning universal image-text representations.arXiv preprint arXiv:1909.11740,

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations.arXiv preprint arXiv:1909.11740,

Pith/arXiv arXiv 1909
[3]

An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Pith/arXiv arXiv 2010
[7]

Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

Pith/arXiv arXiv
[9]

Learning transferable visual models from natural language supervision

11 Published as a conference paper at ICLR 2026 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pp. 8748–8763. PmLR,

2026
[12]

Table 4: The comparisons of image-text retrieval for SEPS-Vit and SEPS-Swin with different selec- tion ratioρon Flicker30K

! 94.6 99.984.9 97.3 72.9 91.757.1 81.3 SEPS ! 95.898.495.1 98.1 86.5 91.7 79.3 95.8 13 Published as a conference paper at ICLR 2026 B COMPARISON OF IMAGE-TEXT RETRIEVAL PERFORMANCE FORSEPS WITH DIFFERENT HYPERPARAMETERS ONFLICKR30K. Table 4: The comparisons of image-text retrieval for SEPS-Vit and SEPS-Swin with different selec- tion ratioρon Flicker30K....

2026
[13]

The analysis reveals both consistencies and notable distinctions in the performance trends across the two architectures. The ViT architecture exhibits a performance curve that is relatively sensitive to the value ofρ, where the rSum score reaches a distinct peak aroundρ= 0.5before declining at a comparatively rapid rate. In contrast, the Swin architecture...

2026
[2015]

Image-question-answer synergistic network for visual dialog

10 Published as a conference paper at ICLR 2026 Dalu Guo, Chang Xu, and Dacheng Tao. Image-question-answer synergistic network for visual dialog. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10434–10443,

2026
[2017]

The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712,

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712,

Pith/arXiv arXiv
[2020]

Vse++: Improving visual- semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

Pith/arXiv arXiv
[2021]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

2019
[2022]

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

12 Published as a conference paper at ICLR 2026 Appendix to “SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment” In this appendix, we provide the following materials: A Comparison of VLP models on Flickr30K and MS-COCO (referring to Section 4.4 in the main paper); B Comparison of image-text retrieval performance for SE...

2026
[2023]

Novel cross- dimensional coarse-fine-grained complementary network for image-text matching.PeerJ Com- puter Science, 11:e2725, 2025a

Meizhen Liu, Anis Salwa Mohd Khairuddin, Khairunnisa Hasikin, and Weitong Liu. Novel cross- dimensional coarse-fine-grained complementary network for image-text matching.PeerJ Com- puter Science, 11:e2725, 2025a. Yang Liu, Wentao Feng, Zhuoyao Liu, Shudong Huang, and Jiancheng Lv. Aligning informa- tion capacity between vision and language via dense-to-sp...

Pith/arXiv arXiv
[2024]

In stark contrast, SEPS not only significantly surpasses all prior fine-grained methods but also successfully bridges this performance gap

demonstrate strong performance on the CLIP backbone, they do not consistently close the performance gap to leading VLP models such as BLIP (Li et al., 2022), particularly on the more challenging MS-COCO benchmark. In stark contrast, SEPS not only significantly surpasses all prior fine-grained methods but also successfully bridges this performance gap. Not...

2022

[1] [1]

Uniter: Learning universal image-text representations.arXiv preprint arXiv:1909.11740,

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations.arXiv preprint arXiv:1909.11740,

Pith/arXiv arXiv 1909

[2] [3]

An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Pith/arXiv arXiv 2010

[3] [7]

Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

Pith/arXiv arXiv

[4] [9]

Learning transferable visual models from natural language supervision

11 Published as a conference paper at ICLR 2026 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pp. 8748–8763. PmLR,

2026

[5] [12]

Table 4: The comparisons of image-text retrieval for SEPS-Vit and SEPS-Swin with different selec- tion ratioρon Flicker30K

! 94.6 99.984.9 97.3 72.9 91.757.1 81.3 SEPS ! 95.898.495.1 98.1 86.5 91.7 79.3 95.8 13 Published as a conference paper at ICLR 2026 B COMPARISON OF IMAGE-TEXT RETRIEVAL PERFORMANCE FORSEPS WITH DIFFERENT HYPERPARAMETERS ONFLICKR30K. Table 4: The comparisons of image-text retrieval for SEPS-Vit and SEPS-Swin with different selec- tion ratioρon Flicker30K....

2026

[6] [13]

The analysis reveals both consistencies and notable distinctions in the performance trends across the two architectures. The ViT architecture exhibits a performance curve that is relatively sensitive to the value ofρ, where the rSum score reaches a distinct peak aroundρ= 0.5before declining at a comparatively rapid rate. In contrast, the Swin architecture...

2026

[7] [2015]

Image-question-answer synergistic network for visual dialog

10 Published as a conference paper at ICLR 2026 Dalu Guo, Chang Xu, and Dacheng Tao. Image-question-answer synergistic network for visual dialog. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10434–10443,

2026

[8] [2017]

The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712,

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712,

Pith/arXiv arXiv

[9] [2020]

Vse++: Improving visual- semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

Pith/arXiv arXiv

[10] [2021]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

2019

[11] [2022]

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

12 Published as a conference paper at ICLR 2026 Appendix to “SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment” In this appendix, we provide the following materials: A Comparison of VLP models on Flickr30K and MS-COCO (referring to Section 4.4 in the main paper); B Comparison of image-text retrieval performance for SE...

2026

[12] [2023]

Novel cross- dimensional coarse-fine-grained complementary network for image-text matching.PeerJ Com- puter Science, 11:e2725, 2025a

Meizhen Liu, Anis Salwa Mohd Khairuddin, Khairunnisa Hasikin, and Weitong Liu. Novel cross- dimensional coarse-fine-grained complementary network for image-text matching.PeerJ Com- puter Science, 11:e2725, 2025a. Yang Liu, Wentao Feng, Zhuoyao Liu, Shudong Huang, and Jiancheng Lv. Aligning informa- tion capacity between vision and language via dense-to-sp...

Pith/arXiv arXiv

[13] [2024]

In stark contrast, SEPS not only significantly surpasses all prior fine-grained methods but also successfully bridges this performance gap

demonstrate strong performance on the CLIP backbone, they do not consistently close the performance gap to leading VLP models such as BLIP (Li et al., 2022), particularly on the more challenging MS-COCO benchmark. In stark contrast, SEPS not only significantly surpasses all prior fine-grained methods but also successfully bridges this performance gap. Not...

2022