{"id":"2b5372b3-00b5-48c7-a1a2-724e6222621d","arxiv_id":"2505.21973","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TSAM combines token-level fusion of visual and textual data with structure-anchored contrastive learning, outperforming prior multi-modal KGC models on DB15K, MKG-W, and MKG-Y.","lead":"This paper presents TSAM, a model for multi-modal knowledge graph completion that combines token-level image and text fusion with contrastive losses that anchor other modalities to graph structure. The authors report state-of-the-art results on three standard benchmarks, with the largest gains on DB15K and MKG-W.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing-modality handling (§III-C/D vs Table I) is unspecified; if absent visual/text entities are excluded or padded differently from baselines, the reported MRR gains may be inflated.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing gap. The central claim is an empirical SOTA improvement, and the comparison's validity depends on all entities being scored under the same protocol. Table I shows that a non-negligible fraction of entities in MKG-W/Y lack visual or textual data; Section III-C provides tokenizers only for entities that have such data, and Section III-D's contrastive losses require paired embeddings for every batch element. Without an explicit missing-modality policy, the reported numbers could be affected in either direction, and the comparison against MyGO/SNAG/NativE is not fully controlled. A secondary concern is the unclear connection between the decoder output t_p (Eq. 20) and the KGE scoring function in the loss (Eqs. 21–22); the reader noted this in the rationale. I did not select it as the primary concern because it is primarily a reproducibility/clarity issue that can be resolved by reading the code, whereas the missing-modality handling can change the evaluation set itself and thus directly alter the headline numbers. If the concrete check shows no distortion, the central claim stands; if it shows exclusion or different padding, the comparison is unfair and the verdict should move toward REJECT. For now, the reader's CONDITIONAL verdict is the right call, so no adjustment is needed.","tokens_in":22159,"tokens_out":9012,"duration_ms":95847,"concrete_test":"Inspect the released code (github.com/2391134843/TSAM) and identify: (1) what token sequence/embedding is produced for entities with no image or no description, (2) whether such entities contribute to L_SV/L_ST, and (3) whether they remain in the candidate entity set during validation/test ranking. Then rerun the MKG-W and MKG-Y evaluations with the missing-modality convention used by MyGO (reference [25])—e.g., same placeholder image/text—while keeping all 15,000 entities in the candidate set. If MRR or Hits@1 shifts by more than ~1 percentage point on either dataset, the reported 7.43%/1.5% improvements are not robust to the missing-modality specification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table I reports MKG-W with 14,463/15,000 entities having visual data and 14,123/15,000 having text (MKG-Y: 14,244/15,000 and 14,305/15,000; DB15K: 12,818/12,842 visual), yet Section III-C defines tokenizers only for 'the image corresponding to each entity' (Eq. 1) and 'the text description paragraph corresponding to each entity' (Eq. 2). Section III-D then defines contrastive losses L_SV and L_ST over positive pairs (S_i,V_i) and (S_i,T_i) for every entity in a batch (Eqs. 13–19), and the fusion step (Eqs. 10–11) requires e_vis and e_txt for all entities. No mechanism is given for entities lacking one or both auxiliary modalities: no placeholder token, no learned 'missing' embedding, no exclusion rule, and no statement of whether such entities appear in the evaluation candidate list. If the released implementation drops these entities from the contrastive loss or from the ranking candidates, the reported MRR/Hits@1 improvements over MyGO/SNAG/NativE on MKG-W/Y would be inflated, because the hardest (modality-incomplete) entities would be removed from evaluation. This is the most load-bearing unresolved assumption for the central SOTA claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSAM, a multi-modal knowledge graph completion model combining (i) Fine-grained Modality Awareness Fusion (FgMAF), which tokenizes visual and textual entity data with BEIT-V2 and BERT, encodes the tokens with transformer encoders, and fuses the resulting embeddings with structural KGE embeddings under a learned attention mechanism, and (ii) Structure-aware Contrastive Learning (SaCL), which adds two contrastive losses that align visual and textual embeddings with structural embeddings. The model is trained with a cross-entropy link-prediction loss plus the two contrastive losses. Experiments on DB15K, MKG-W, and MKG-Y report improvements over published baselines, with ablations attributing the gains mainly to SaCL and FgMAF.","tokens_in":22461,"tokens_out":5495,"duration_ms":52262,"significance":"The central claim, that structure-anchored contrastive learning improves MMKGC by reducing modality noise while preserving graph structure, is plausible and potentially useful for the field. The paper provides a motivating demonstration in Fig. 2 that existing models degrade sharply without structural modality knowledge, and the ablations in Table III consistently show that SaCL contributes to the reported performance. The reported MRR improvements on DB15K and MKG-W are sizable. The paper also ships code, includes a parameter sensitivity study, and reports experiments with different decoders and token counts. However, the empirical significance is currently qualified by an underspecified treatment of entities with missing modalities, an unconnected decoder output in Section III.E, an ambiguous fusion operation in Eq. (10), and a baseline comparison that mixes reproduced and source-paper numbers without variance reporting.","major_comments":[{"comment":"The model is underspecified for entities that lack visual or textual data. Table I shows that MKG-W has 14,463 of 15,000 entities with visual data and 14,123 with text, and MKG-Y has similar gaps, yet Eqs. (1)-(2) define tokenizers only for images and text descriptions that exist, Eqs. (10)-(11) require e_vis and e_txt for every entity, and Eqs. (13)-(19) form positive contrastive pairs (S_i,V_i) and (S_i,T_i) for every batch element. The paper must specify what token sequence or embedding is used for entities with missing modalities, how the contrastive losses treat such pairs, and whether evaluation candidates are restricted to entities with all modalities. Without this, the reported gains over SNAG, NativE, and MyGO on MKG-W/Y could be inflated by removing or down-weighting the hardest modality-incomplete entities from ranking or contrastive training.","section":"§III-C/III-D and Table I"},{"comment":"The decoder output t_p is never connected to the scoring function used in the prediction loss. Eq. (20) defines t_p = g_d([CLS], h_f, r), but Eqs. (21)-(22) define the loss through Theta(h,r,t_n) = sigmoid(Score(h,r,t_n)), with Score taken from the KGE scoring functions in Eqs. (7)-(9). No equation states whether Score is computed with the fused representation e_f, the structural embedding e_str, or t_p, nor how t_p is compared with candidate entities to produce a distribution over |E| tails. Please specify the exact training and inference scoring pipeline, including how t_p is converted into scores over candidate entities.","section":"§III.E, Eqs. (20)-(22)"},{"comment":"The fusion operation is not well-defined. Eq. (10) says e_f = stack(alpha_s e_str, alpha_v e_vis, alpha_t e_txt), while Eq. (11) computes the attention weights from dot products involving alpha and the three embeddings. It is unclear whether e_f is a concatenation of three weighted vectors, a weighted sum, or some other tensor operation, and how the attention vector alpha is learned. This matters because the proposed modality-awareness mechanism could amount to no more than per-modality scaling if the operation is not specified. Please define the exact tensor operation and dimensions, and clarify how the attention weights are trained.","section":"§III.C, Eqs. (10)-(11)"},{"comment":"The state-of-the-art comparison mixes reproduced results (only SNAG and NativE are marked with a reproduced symbol) with numbers taken from source papers and prior reports, and no variance, standard deviation, or repeated-seed statistics are reported. Since the central claim is an empirical improvement of 7.37% MRR on DB15K and 7.43% on MKG-W, the paper should report mean and variance over multiple seeds, specify the exact evaluation protocol including candidate filtering and handling of missing modalities, and ideally re-run all baselines under the same protocol or justify why cross-paper numbers are directly comparable.","section":"Table II and §IV.B"}],"minor_comments":[{"comment":"The MRR formula uses |E| and then |N| inconsistently in the same displayed equation; the denominator should be the number of test triples, not the entity set size, and the notation should be unified.","section":"Eq. (24)"},{"comment":"Line 19 says the prediction loss is L_ST, but L_ST is later defined as the structure-text contrastive loss; the prediction loss is L_p in Eq. (23). Please correct this label.","section":"Algorithm 1, line 19"},{"comment":"The textual token projection in Eq. (4) writes the result as {t'_1,...,t'_n}, but Eq. (2) defines m textual tokens; the index should be m. Please also state the padding and truncation strategy for token sequences, which interacts with the missing-modality issue.","section":"§III.C, Eq. (4)"},{"comment":"The paper sometimes refers to two datasets and sometimes to three, and dataset statistics are called 'two datasets' in Section IV.A. In Table II, the SNAG row for MKG-Y has '41.0846.70' without separation, and several model names (MyGO/MyGo, SNAG/SnAg, NativE/Native) are used inconsistently. Please standardize these textual details.","section":"§IV and Table II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own prior work [25] for tokenization and baseline numbers, and the missing-modality handling may be present in the released code but is not described in the paper. The central idea is worth pursuing, but the load-bearing gaps in the model specification and evaluation protocol need to be addressed before the SOTA claim can be accepted. The paper's scope is appropriate for a multimedia journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TSAM is a credible, incremental MMKGC paper whose central empirical claim is probably right but not fully supported as written. The genuinely new piece is SaCL, the structure-anchored contrastive losses that pull visual and textual embeddings toward the structural embedding, plus the token-level fusion (FgMAF) that goes beyond MyGO's concatenation. The ablations show both components matter, and the gains over MyGO/SNAG on DB15K and MKG-W are consistent across MRR and Hits@N. That is real work, and the architecture is a reasonable next step in a line the authors know well. Credit where due: the code link is given, and the ablation design covers the main components.\n\nNow the soft spots. The stress-test concern is the one I would lead with: Table I shows small but nonzero missing-modality counts on all three datasets, yet Sections III-C and III-D define tokenization, fusion, and contrastive losses as if every entity has both an image and a text. There is no placeholder token, no missing-modality embedding, no exclusion rule, and no statement about whether such entities appear in the ranking candidate list. If the released code drops these entities from the contrastive loss or from evaluation, the reported MRR improvements could be inflated. The missing rates are small (about 3-6%), so the effect may be modest, but the paper gives no evidence either way. That is the single most important thing to fix.\n\nSecond, the decoder output t_p in Eq. (20) is never connected to the scoring functions in Eqs. (7)-(9) or to Eq. (22). It appears in Algorithm 1, but the formal link is missing. Probably just a write-up gap, but reviewers will need it spelled out. Third, the evaluation hygiene is weak: no error bars or repeated runs, only two baselines reproduced (SNAG and NativE), most numbers taken from other papers or the authors' own prior work. For a paper claiming SOTA, that is standard practice in this subfield but still a limitation. The MKG-Y gains are small (1.5% MRR), so the 'consistent improvement' claim rests mostly on DB15K and MKG-W. The citation list has minor errors, including NativE referred to with inconsistent numbers and AdaMF's reference mismatched; these are fixable.\n\nOverall, the argument is coherent, the ablation supports the design, and I do not see a load-bearing flaw. The missing-modality issue is not reason to desk-reject; it is reason to ask for clarification and a reproducibility pass. Send it to review, but this should be conditional on the authors explaining how entities with missing images or text are handled and ideally showing results with and without those entities. I would bring it to a reading group for the contrastive alignment idea, and I would probably cite it if the code checks out.","headline":"TSAM is a sensible, incremental extension of MyGO with plausible SOTA results on two datasets, but the missing-modality handling is unspecified and the evaluation needs more rigor before the gains are trusted.","tokens_in":834,"tokens_out":1752,"would_cite":true,"duration_ms":36154,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structure-anchored contrastive learning lifts multi-modal KG completion by up to 7.4% MRR.","keywords":["multi-modal knowledge graph completion","fine-grained modality fusion","structure-aware contrastive learning","knowledge graph embedding","link prediction","visual-textual tokenization","modality attention","contrastive loss alignment"],"falsifier":"Run TSAM on MKG-W and MKG-Y with all entities that lack an image or a text description removed from the evaluation set, and also with those entities replaced by a learned placeholder token; if the MRR gap over MyGO shrinks or disappears under either variant, the reported gains come from missing-modality handling rather than from structure-aware contrastive learning.","tokens_in":21960,"feed_emoji":"🧩","tokens_out":7574,"duration_ms":66294,"temperature":0.7,"pith_summary":"This paper sets out to show that multi-modal knowledge graph completion works best when graph structure, not images or text, is the anchor of entity representations. To that end it proposes TSAM, which fuses image and text information at the level of individual tokens and applies two contrastive losses that pull visual and textual embeddings toward the structural embedding of each entity. On the DB15K, MKG-W, and MKG-Y benchmarks, TSAM reports mean reciprocal rank gains of 7.37%, 7.43%, and 1.5% over the strongest baseline, with larger gains on Hits@1 than on Hits@10. The practical point is that modality noise can be tamed by explicitly subordinating auxiliary modalities to graph structure.","feed_headline":"Graph structure first lifts multi-modal KG completion by 7.4%","feed_subtitle":"TSAM's contrastive losses pull visual and textual embeddings toward graph structure, cutting fusion noise.","key_machinery":"The two load-bearing mechanisms are FgMAF and SaCL. FgMAF is a fine-grained modality fusion pipeline: visual and textual tokenizers turn each entity's image and description into discrete token sequences, linear projections map both into one space, a Transformer encoder pools them into embeddings, and a softmax attention over structural, visual, and textual embeddings produces a fused entity representation. SaCL is the structure-anchoring component: it computes two symmetric contrastive losses, one between structural and visual embeddings and one between structural and textual embeddings, with in-batch negatives, so the fused representation is regularized to stay close to the graph-structure embedding. A knowledge graph embedding scoring function (TuckER, TransE, or RotatE) supplies structural embeddings and scores candidate triples during training.","core_discovery":"TSAM claims that the two obstacles to multimodal knowledge graph completion are coarse modality fusion and neglect of structural dominance. Its Fine-grained Modality Awareness Fusion (FgMAF) tokenizes images with BEIT-V2 and descriptions with BERT, linearly projects both token sequences into a shared space, encodes them with a Transformer, and then weights the resulting visual, textual, and structural embeddings with an attention vector before decoding a tail entity. Its Structure-aware Contrastive Learning (SaCL) adds symmetric contrastive losses between structural and visual embeddings and between structural and textual embeddings, so that auxiliary modalities are brought closer to the structure modality rather than allowed to introduce noise. With these components, TSAM reports the best MRR, Hits@1, Hits@3, and Hits@10 on DB15K and MKG-W and the best MRR on MKG-Y, with ablation showing that removing SaCL costs more than removing FgMAF.","pith_inferences":["An implication the paper leaves implicit is that the same structure-anchored contrastive loss could be dropped into other fusion architectures, potentially improving their robustness to noisy or irrelevant images and text without retraining their tokenizers.","The paper does not describe how entities lacking an image or a description are handled; if missing modalities are zero-padded, a natural test is to compare against masked or learned-placeholder alternatives before attributing gains to contrastive alignment.","A testable extension is to replace the fixed attention weights with per-entity modality credibility scores, which the paper lists as future work but does not evaluate; this could matter on datasets with more missing or corrupted modalities than DB15K and MKG-W.","Because TSAM centers contrastive learning on structure, it suggests that incremental structure-aware contrastive updates, rather than full retraining, may let multimodal knowledge graph completion models adapt to dynamic knowledge graphs; the paper names this direction but does not test it."],"forward_implications":["If TSAM's central claim is right, future multimodal knowledge graph completion models should treat graph structure as the reference modality and align images and text to it, rather than fusing all modalities symmetrically.","Fine-grained token-level interaction between image and text tokens is worth more than whole-embedding concatenation, because removing FgMAF lowers MRR on both main benchmarks.","Structure-aware contrastive alignment improves ranking quality most at the top of the list, since Hits@1 gains exceed Hits@10 gains on DB15K and MKG-W.","The choice of structural scoring function matters: TuckER gives the best fused result on MKG-W, ahead of RotatE and TransE, so the structure encoder remains a significant source of performance."],"supporting_citations":[{"why":"provides the fine-grained visual tokenization setting and the strongest token-based baseline TSAM extends and compares against.","marker":"[25]"},{"why":"supplies the visual encoder that converts entity images into discrete visual tokens.","marker":"[27]"},{"why":"supplies the text tokenizer and the Transformer encoder/decoder backbone.","marker":"[30]"},{"why":"provides the TuckER scoring function used as the structural encoder in the main TSAM configuration.","marker":"[34]"},{"why":"provides one of the structural scoring functions compared in the scoring-function ablation.","marker":"[14]"},{"why":"provides the RotatE structural scoring function compared in the scoring-function ablation.","marker":"[15]"},{"why":"is the DB15K benchmark dataset on which TSAM reports its largest gains.","marker":"[57]"},{"why":"is the MKG-W and MKG-Y benchmark dataset source used for the other two evaluation sets.","marker":"[51]"},{"why":"supplies the contrastive learning formulation that SaCL adapts to align visual and textual embeddings with structural embeddings.","marker":"[54]"}],"fun_headline_variants":["Contrastive pull tames modal noise for KG completion","SaCL beats FgMAF in multi-modal KG completion","Structure-aware contrastive learning lifts multi-modal KG completion","TSAM: align images and text to graph structure for KG completion","Structure-first multi-modal KG completion beats noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every entity can be given a meaningful visual or textual token sequence, but the paper never states what is fed into the model for the roughly 4 to 6 percent of entities on MKG-W and MKG-Y that lack an image or a description, so the reported comparison could hinge on that unspecified handling.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive pull tames modal noise for KG completion","SaCL beats FgMAF in multi-modal KG completion","Structure-aware contrastive learning lifts multi-modal KG completion","TSAM: align images and text to graph structure for KG completion","Structure-first multi-modal KG completion beats noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000754,"raw_usage":{"total_tokens":3369,"prompt_tokens":976,"completion_tokens":2393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2313}},"tokens_in":592,"tokens_out":2393,"duration_ms":17039,"temperature":1.0,"reasoning_tokens":2313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:18:28.033861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TSAM on MKG-W and MKG-Y with all entities that lack an image or a text description removed from the evaluation set, and also with those entities replaced by a learned placeholder token; if the MRR gap over MyGO shrinks or disappears under either variant, the reported gains come from missing-modality handling rather than from structure-aware contrastive learning.","supporting_citations":[{"cited_title":"Mmkg: multi-modal knowledge graphs,","cited_arxiv_id":null,"evidence_quote":"is the DB15K benchmark dataset on which TSAM reports its largest gains."},{"cited_title":"Relation-enhanced negative sampling for multimodal knowledge graph completion,","cited_arxiv_id":null,"evidence_quote":"is the MKG-W and MKG-Y benchmark dataset source used for the other two evaluation sets."}],"review_version":1}