REVIEW 4 major objections 4 minor 2 references
Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that separating the text prompt into hard pair, soft attribute, and soft object prompts, then fusing text and image features first across modalities and then within modalities, improves compositional zero-shot recognition…
desk verdict The architecture is plausible but the empirical core is unverifiable: baseline tables are corrupted and prose gains don't match the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing structure is the set of three prompt-conditioned text features and their corresponding image features. A hard pair prompt $P_{hard}^{pair}=\{a\,photo\,of,x_s,x_o\}$ feeds both state and object labels at once, while a soft attribute prompt $P_{soft}^{attr}=\{x_1,\dots,x_k,x_s\}$ and a soft object prompt $P_{soft}^{obj}=\{x_1,\dots,x_k,x_o\}$ each prepend learnable tokens to a single label. The Modal Fusion Synthesizer Block (MFSB) applies cross-attention twice: inter-modality fusion alternates text and image features as query, key, and value to merge the two modalities, and intra-modality fusion then uses the attribute feature and object feature of the same modality as key and value for each other. The losses from the pair, attribute, and object branches at the initial, inter-fused, and intra-fused stages are combined into a total loss with weights $\alpha=\beta=\gamma=0.2$. This machinery is what lets the attribute and object each keep their own text representation while forcing the image and text sides to refine each other twice, which the paper argues captures subtle semantic differences.
What would settle it
A re-implementation that runs the paper's exact training recipe (CLIP ViT-L/14, 20 epochs, Adam, same splits and loss weights) and the DFSP baseline under identical conditions would settle the claim: if DFSP reaches or exceeds the paper's reported HM and AUC values on MIT-States and UT-Zappos, the separated-prompt margin is not reproducible. A cheaper check is to verify whether the Co-CGE row in the closed-world table, which duplicates the open-world Co-CGE row, matches the originally published Co-CGE numbers.
Extended reading notes
Core claim
The core discovery the paper tries to establish is that prompt decomposition changes what a vision-language model can extract from an image-text pair. Instead of feeding a single pair prompt 'a photo of [state] [object]' into the CLIP text encoder, the method keeps that hard pair prompt but adds two learnable soft prompts, one for the state and one for the object. The resulting text features and the image features are passed through a Modal Fusion Synthesizer Block: first a cross-attention layer mixes text and image features for each prompt (inter-modality fusion), then a second cross-attention layer lets the state feature and object feature exchange information of the same type (intra-modality fusion). The paper argues that this two-stage, separated-prompt design outperforms the single-prompt baseline and several prior CZSL methods, and that the best configuration is hard pair, soft object, soft attribute with inter-fusion before intra-fusion.
Load-bearing premise
The entire claimed advantage is measured against baseline numbers in Tables 1 and 2, so the central claim collapses if those numbers were transcribed incorrectly or if the comparison protocol differed between methods.
Editorial extensions
If this is right
- On the three benchmarks used, the method reports the best or near-best seen, unseen, harmonic mean, and AUC scores in both open-world and closed-world settings, making it the stated new reference point among the compared CZSL methods.
- The ablation tables show that all three prompt components contribute: pair-only is the weakest configuration, and adding separated object and attribute prompts raises harmonic mean and AUC on MIT-States.
- The fusion-order ablation identifies inter-modality fusion followed by intra-modality fusion as the best arrangement, giving future systems a concrete ordering rule for cross-attention fusion blocks.
- Because the added components are only prompts and cross-attention layers wrapped around a frozen CLIP encoder, the method can be grafted onto any CLIP-based CZSL pipeline without retraining the backbone.
Reading between the lines
- A natural testable extension is to apply the same separated-prompt and two-stage fusion recipe to other CLIP-based composition tasks, such as human-object interaction detection, where the paper's ablation suggests the pattern would transfer; the paper does not run those experiments.
- Because the prompt-form search in Table 3 is only run on MIT-States, the claim that hard pair plus soft object plus soft attribute is universally the best configuration across datasets is an extrapolation the paper has not directly verified.
- If the loss weights $\alpha=\beta=\gamma$ were made learnable instead of fixed at 0.2, the method might adaptively balance the initial, inter-fused, and intra-fused losses in open-world versus closed-world settings; this is a modification the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a compositional zero-shot learning (CZSL) method that separates prompts into a hard pair prompt and soft attribute/object prompts, then fuses the resulting vision-language features through a Modal Fusion Synthesizer Block (MFSB) with inter-modality followed by intra-modality cross-attention. The central claim is that this decomposition and fusion order improves CZSL accuracy over existing baselines including DFSP, on MIT-States, UT-Zappos, and C-GQA in both open-world and closed-world settings. The paper reports state-of-the-art or near-state-of-the-art results in Tables 1 and 2, supported by three ablations (prompt type, prompt components, fusion order).
Significance. If the reported results were reliable, the contribution would be a modest but useful incremental improvement over DFSP, with a fairly thorough ablation study of prompt types and fusion order. The idea of separating hard pair prompts from soft attribute and object prompts, and the specific inter-then-intra fusion order, is a reasonable design choice that could interest the CZSL and prompt-learning communities. However, the empirical evidence is the only support for the central claim, and that evidence is internally inconsistent: two baseline rows in Table 2 are either identical across open/closed settings or repeated across datasets, and the percentage gains quoted in Section 4.2 do not match the table values. Because the paper's contribution is empirical and the baseline table appears corrupted, the significance cannot be assessed until the numbers are corrected and verified.
major comments (4)
- [Tables 1 and 2] The Co-CGE row in Table 2 is numerically identical to the Co-CGE row in Table 1 for all three datasets and all four metrics (e.g., MIT-States: 31.1, 5.8, 6.4, 1.1). This cannot be correct because open-world and closed-world evaluations use different seen/unseen splits, so the same baseline cannot yield identical Seen, Unseen, HM, and AUC values in both settings. Likewise, the CGE row in Table 2 lists the same four values (28.7, 25.3, 17.2, 5.1) for MIT-States and C-GQA. These are copy-paste artifacts that call into question the integrity of the entire baseline comparison. Since the central claim of superiority over DFSP and other methods rests entirely on the margins in these tables, the paper's main result is currently unverifiable.
- [Section 4.2, Results] The text claims: "Ours outperforms in AUC scores of 5.4% on MIT-States, 36.0% on UT-Zappos, and 10.5% on CGQA, surpassing by 4.3%" and "6.8% increase in the harmonic mean on the MIT-States." No consistent interpretation of Table 1 yields these numbers. For example, UT-Zappos AUC in Table 1 is 29.27 (Ours) vs. 26.4 (DFSP), which is a relative improvement of about 10.9%, not 36.0%. MIT-States HM is 20.35 vs. 19.1, a relative gain of about 6.5%, not 6.8%. The claimed percentages are unsupported by the tables; the reader cannot reconstruct the reported gains, which is a serious problem for a results section.
- [Section 3.5 and Table 3] The final prompt configuration (Hard Pair, Soft Object, Soft Attribute) and the loss weights alpha=beta=gamma=0.2 appear to be selected from the 27 runs in Table 3, which are all evaluated on the MIT-States open-world benchmark. The paper does not describe a held-out validation set or any procedure for choosing hyperparameters without using the test split. Consequently, the reported final numbers may partially reflect test-set information, and the claimed improvements over baselines that do not receive such tuning are not fairly compared. This is a load-bearing methodological issue for the paper's empirical claim.
- [General reproducibility] No code, random seeds, or implementation details of the cross-attention layers (e.g., number of heads, dimension of Q/K/V, attention dropout) are provided. The loss combination in Section 3.5 uses unclear notation (e.g., L_pair_soft, L_attr_soft, L_obj_soft are referenced but not all defined). Without these details, the ablations in Tables 3-5 cannot be reproduced independently, further reducing the confidence in the reported margins.
minor comments (4)
- [Section 4.1] The dataset description mixes up UT-Zappos and MIT-States: after describing MIT-States, the text says "The dataset has 50,025 shoe images labeled under 12 categories and 16 attributes," which is the UT-Zappos description, but appears in the MIT-States paragraph. This should be corrected.
- [References] Several references are garbled or duplicated. For example, references [10], [11], [12], and [13] all appear to describe the same 'Learning graph embeddings for compositional zero-shot learning' paper with different venue strings. Reference [1] is misattributed to Apley and Zhu instead of the intended CZSL work. The reference list needs a thorough revision.
- [Notation] The notation in Sections 3.4.1 and 3.4.2 is inconsistent: 'inter-fused' and 'intra-fused' features share superscripts that are easy to confuse (e.g., v_pair_intra1 vs. v_attr_intra2). The definitions of losses like L_pair_soft are not given explicitly; they are introduced only in the total loss equation. Please define all terms before use.
- [Figures] Figure 1 and Figure 2 are referenced but not included in the text; the captions suggest they are important for understanding the fusion layout. If the figures are present in the actual submission, ensure they are visible and legible; otherwise, the text alone is insufficient to follow the architecture.
Circularity Check
Architecture is not circular, but the headline MIT-States open-world result is the argmax of the paper's own ablation tables evaluated on the same benchmark.
-
fitted input called prediction
[Section 4.3, Tables 3 and 5; Section 4.1, Table 1 (Open-World, MIT-States row).]
"Every experiment is conducted with the MIT-States dataset in the Open-world setting. ... Hard {Pair}, Soft {Obj}, Soft {Attr} 49.33 19.01 20.35 7.33 ... 1. Inter 2. Intra 49.33 19.01 20.35 7.33"
The paper selects the prompt configuration and fusion order from 27 ablation runs and 2 fusion orders on the MIT-States open-world setting, then reports the selected row as the method's own performance. Table 1 lists Ours on MIT-States as 49.33/19.01/20.35/7.33, exactly the Table 3 row 'Hard {Pair}, Soft {Obj}, Soft {Attr}' and the Table 5 row '1. Inter 2. Intra' with identical numbers. Because those ablations are the evaluation set used for the headline result and no held-out validation split is described, the reported improvement is the maximum of a search over the test benchmark rather than an independent prediction. This is a partial reduction by construction: that particular claim is equivalent to its own selection criterion.
full rationale
No formal derivation loop, self-citation chain, or uniqueness theorem appears. The method's components (hard pair prompt, soft attribute/object prompts, inter- then intra-modality cross-attention) are defined independently of the benchmarks, and the improvements on UT-Zappos, C-GQA, and closed-world settings are empirical comparisons rather than algebraic consequences of the inputs. The only load-bearing reduction-by-construction I can exhibit is the MIT-States open-world headline: the final Ours row equals the best rows of Tables 3 and 5, and Table 3 states all ablations used the MIT-States open-world setting, so the best configuration and fusion order were selected on the same benchmark reported in Table 1. That is selection-based circularity (test-set information entering model selection), not a derivation from fitted parameters. I did not count the table inconsistencies (Co-CGE identical across open/closed worlds in Tables 1-2; CGE's C-GQA row equal to its MIT-States row; Section 4.2 percentage gains not matching table values) as circularity; they are correctness/verification risks. Overall score 4 reflects one partially circular headline claim while the central architecture retains independent content elsewhere.
Assumptions & free parameters
free parameters (3)
- Loss weights alpha, beta, gamma =
0.2 each
- Prompt type configuration =
Hard pair + Soft object + Soft attribute
- Soft prompt learnable tokens =
not reported
assumptions (4)
- domain assumption Pre-trained CLIP ViT-L/14 encoders embed images and text into a shared space where separated attribute and object prompts carry semantically meaningful features.
- domain assumption Cross-attention between features from different prompts (e.g., object text as key/value for attribute queries) produces features that help zero-shot generalization.
- domain assumption Standard supervised training with cross-entropy on seen compositions transfers to unseen compositions.
- domain assumption Benchmark splits and evaluation protocols from prior datasets are used correctly and are comparable.
Cite this review
Pith. "Pith review of Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learning." pith.science (2026). https://pith.science/paper/JCAXMLWW
@misc{pith2026250117171,
author = {Pith},
title = {Pith review of: Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/JCAXMLWW}},
note = {Machine review of arXiv:2501.17171}
}
read the original abstract
Compositional Zero-Shot Learning (CZSL) aims to recognize subtle differences in meaning or the combination of states and objects through the use of known and unknown concepts during training. Existing methods either focused on prompt configuration or on using prompts to tune the pre-trained Vision-Language model. However, these methods faced challenges in accurately identifying subtle differences in meaning or combining states with objects. To jointly eradicate the above issues and construct an efficient and effective CZSL technique, we suggest a method to improve attribute recognition performance by utilizing diverse Prompt Learning with an Inter/Intra-Modality Fusion Synthesizer in scene understanding involving subtle semantic differences and multiple objects.
Figures
Reference graph
Works this paper leans on
-
[17]
Learning to compose soft prompts for compositional zero-shot learning
Peilin Yu Nihal V Nayak and Stephen H Bach. Learning to compose soft prompts for compositional zero-shot learning. In arXiv preprint arXiv:2204.03574,
-
[24]
On transferability of prompt tuning for natural language processing
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...
work page 2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.