{"id":"e35f7de8-620a-44a8-a9af-1e27b185a786","arxiv_id":"2505.07895","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HGNN-IMA adaptively blends per-modality attention during graph propagation and improves node classification on five multi-modal heterogeneous network benchmarks by 0.3 to 2.5 Macro-F1 points.","lead":"The paper proposes a graph neural network that lets text, image, and other modalities of a node influence how information flows between nodes, and tests it on movie and product networks. It reports small but consistent gains in node classification over earlier fusion methods and releases code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8) alignment modulation is sign-inverted: the softmax over |g_m1 - g_m2| up-weights, not down-weights, modality-misaligned neighbors, contradicting the paper's central mechanism.","rationale":"The reader's weakest assumption is fair baseline tuning and run-to-run noise, which is a legitimate empirical concern. However, the most load-bearing issue in this manuscript is internal: Eq. (8), the only formal definition of the modality-alignment modulation, is sign-inverted relative to the paper's stated purpose. Softmax over positive inconsistency scores assigns larger attention to more inconsistent neighbors, and Eq. (9) multiplies this term into the aggregation weights, so the model would up-weight exactly the nodes the paper says it down-weights. The ablation in Section 5.5, where removing the term hurts, cannot rescue this inconsistency unless the implemented code uses a different formula than the one published. A code-level check will settle the question: if the code follows Eq. (8), the central contribution is not as described; if it uses the inverse, the paper needs a corrected equation. The baseline-tuning concern is secondary: even if all baselines are fairly tuned, the formal specification of a core mechanism remains wrong. I also note in passing that Table 3's t-values are hard to reproduce from the reported means and standard deviations (e.g., DOUBAN mean difference of about 0.0129 with an HGT standard deviation of 0.1261 cannot plausibly yield t=3.01 for five runs), which further weakens the statistical evidence but is not the primary issue. The recommendation stays CONDITIONAL because the concern is addressable and the empirical claim might still hold; the condition is that the authors must correct or justify Eq. (8) with corresponding code evidence.","tokens_in":15366,"tokens_out":10021,"duration_ms":97146,"concrete_test":"","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The formal definition of the alignment modulation in Eq. (8) does the opposite of what the paper claims. It defines bar_beta_ij = softmax over neighbors of sum_{m1,m2} |g_ij^{m1} - g_ij^{m2}|. Since softmax is monotonically increasing in its argument, a neighbor whose modality-specific similarity scores disagree the most receives the largest bar_beta. Eq. (9) then multiplies this into the combined attention beta, so the model amplifies, rather than suppresses, modality-misaligned neighbors. This directly contradicts the stated goal of encouraging propagation among nodes with consistent similarities across all modalities. It also contradicts the Section 5.5 ablation, where removing this modulation (HGNN-IMA-align) hurts performance. If the released implementation follows Eq. (8) literally, one of the three named contributions does not function as described. If the implementation uses exp(-|...|) or a reciprocal, then Eq. (8) is a notational error that must be corrected. Either way, the current text does not support the claimed alignment mechanism, and the explanation of why the model works is unsupported by its own equations.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HGNN-IMA, a heterogeneous graph transformer-based model for node classification in multi-modal heterogeneous networks (MMHNs). The model computes modal-specific similarity scores between nodes, combines them through a nested inter-modal attention mechanism, modulates the combined attention with a term intended to encourage modality alignment, and adds an attention loss to down-weight neighbors whose node types lack certain modalities. Experiments are reported on five datasets (DOUBAN, IMDB, AMAZON, AMAZON-1, AMAZON-2) with comparisons to early-fusion, late-fusion, and other heterogeneous graph baselines, together with ablations and a t-test against the strongest baseline.","tokens_in":15579,"tokens_out":6624,"duration_ms":61093,"significance":"The core idea of fusing modalities inside the GNN propagation process, rather than before or after it, is timely and potentially valuable for multi-modal graph learning. The paper provides a public code repository, evaluates on five datasets, and includes a broad set of ablations that probe the contribution of each component. If the mechanism were correctly specified and the empirical claims were properly supported, this would be a useful contribution. However, the current manuscript contains a load-bearing inconsistency in the alignment-modulation equation and the statistical support for the claimed universal superiority over baselines is incomplete.","major_comments":[{"comment":"The alignment modulation is sign-inverted. As written, ¯β^(k)_ij = softmax_j(Σ_{m1,m2∈M} |g^(k),m1_ij − g^(k),m2_ij|) assigns larger weights to neighbors whose modality-specific similarity scores disagree the most, and Eq. (9) multiplies this into the aggregation weights. This directly contradicts the text, which states that the modulation amplifies the contribution of modality-aligned nodes, and it is also inconsistent with the ablation interpretation in §5.5, where removing this term (HGNN-IMA−align) hurts performance. If the released implementation follows Eq. (8) literally, then one of the three named contributions does not function as described; if the implementation uses exp(−|·|) or a reciprocal, then Eq. (8) is a notational error. Either way, the authors must correct the equation and clarify which form the implementation actually uses; the ablation results cannot be interpreted until this is resolved.","section":"§4.2, Eq. (8)"},{"comment":"The claim that HGNN-IMA 'consistently outperforms all the baselines across all datasets' is not fully supported by the statistics presented. Standard deviations are reported only for HGNN-IMA and HGT; all other baselines appear as point estimates. The Macro-F1 gain over the strongest baseline on AMAZON is roughly 1.2 percentage points (0.8427 vs 0.8302), and without variance information for SeHGNN or other close baselines it is unclear whether these differences exceed run-to-run noise. A two-sample t-test is only conducted against HGT, not against all baselines. Please report standard deviations (or confidence intervals) for every baseline and either conduct significance tests against each baseline or qualify the claim of universal superiority.","section":"§5.4, Table 3"},{"comment":"No hyperparameter tuning procedure is described for the baselines. The authors specify their own settings (learning rate, layer count, dropout, embedding dimension, etc.) but do not state whether each baseline was tuned on the validation set or run with default parameters from its original paper. Since several reported gains over the strongest baseline are small (around one to two Macro-F1 points), undertuned baselines could account for the improvements. Please provide the configuration and tuning protocol for each baseline, including any grid search ranges.","section":"§5.3 and Appendix B"}],"minor_comments":[{"comment":"The reported standard deviation for HGT on DOUBAN is 0.1261, which is two orders of magnitude larger than the other HGT standard deviations and larger than the difference between the model means; this value should be double-checked, as it is unusual for F1 scores across five random seeds.","section":"Table 3"},{"comment":"The sentence 'there are 1.2%, 1.6%, and 0.3% gains in Macro-F1' appears inconsistent with Table 2; for example, the AMAZON gain of 0.8427 vs 0.8302 is about 1.5% relative or 1.2 percentage points, not 0.3%. Please clarify whether these are relative or absolute differences.","section":"§5.4"},{"comment":"The formal derivation uses a single attention head, but §5.3 states that multi-head attention with 8 heads is used during training. The equations should either be extended to the multi-head case or the experimental setting should be reconciled with the formal description.","section":"§4.2, Eq. (6)"},{"comment":"The complexity statement says the worst-case cost is O(|V|^2|M|^2) and then notes that for bounded-degree graphs it is O(|V|n|M|^2); the latter is the practically relevant bound, and the two cases should be presented more clearly to avoid implying that the model is quadratic in the number of nodes in typical networks.","section":"Appendix A"},{"comment":"The preprocessing step of completing missing visual attributes with text features is a strong modeling assumption that is only mentioned in passing; stating it earlier (in the framework description) and ideally providing an ablation over alternative imputation strategies would strengthen the paper.","section":"§5.1 and §4.1"}],"recommendation":"major_revision","confidential_remarks":"The sign inversion in Eq. (8) is the central issue. Since the authors provide code, an obvious verification is to inspect how the alignment modulation is actually implemented; if the code uses a negative exponent or a reciprocal, the paper should be corrected accordingly and the ablation discussion updated. The statistical incompleteness (no variance for most baselines, no tuning protocol) further weakens the headline empirical claim and should be addressed with additional experiments or a more cautious wording. The manuscript is within the scope of the journal, and the proposed architecture is interesting, but the current presentation does not yet support the claimed mechanism or the claimed universal superiority."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real but modest empirical contribution to node classification on multimodal heterogeneous networks, and it has a load-bearing sign error in Equation 8. The paper says the modulation down-weights neighbors with inconsistent modality similarities, but softmax over |g^{m1} - g^{m2}| up-weights them. So the mechanism that is central to the story — and that the ablation in Table 4 says is worth 1–4 points of Macro-F1 — does the opposite of what the text claims. If the released code follows Eq 8 literally, the -align ablation should show an improvement, not a drop. If the code uses exp(-|...|), then Eq 8 is a notational error. Either way the manuscript is internally inconsistent at a key point, and a referee needs to see the code before believing the stated mechanism.\n\nWhat is actually new and good: the nested inter-modal attention in Eq 6 — a pair-specific lambda over modalities applied to HGT's inter-node attention — is a clean idea I have not seen in the cited MHGAT, XGEA, or HGT work. The missing-modality attention loss (Eq 7) is sensible. The experiments cover five datasets, two metrics, five seeds, early stopping, and component ablations. The larger AMAZON-1/2 datasets are a decent stress test, and the code is promised.\n\nSoft spots beyond the sign error: baseline variances are not reported, only HGT's, and that one is alarming on DOUBAN (std 0.1261 on a 0.8758 mean). The t-test is only against HGT, not the other baselines. The 'consistently outperforms' claim is true in the table, but without variance for most baselines I cannot tell how much is noise. The text also misstates the gains (the 0.3% for AMAZON should be about 1.2 points per Table 2). Minor but sloppy.\n\nWho this is for: people working on multimodal graph learning or HGT extensions will find the nested-attention idea worth a look; the sign bug is a caution to check equations against claims. I would not desk-reject this. It deserves a serious referee, and the revision should include a corrected Eq 8 (or a rewritten narrative), full baseline variances, and the code for inspection.\n\nRecommendation: send to peer review, conditionally.","headline":"A genuine nested-attention idea and solid ablation, but Eq 8 inverts the stated alignment mechanism and baseline variance reporting is thin.","tokens_in":16168,"tokens_out":6188,"would_cite":false,"duration_ms":57914,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68R10"],"pacs":[],"model":"deepseek-v4-flash","headline":"HGNN-IMA puts multi-modal fusion inside graph propagation, letting the similarity of two nodes on one modality adjust the attention weight on another, and reports consistent node-classification gains over early- and late-fusion baselines…","keywords":["multi-modal heterogeneous networks","node classification","heterogeneous graph transformer","inter-modal attention","graph neural networks","multi-modal fusion","missing modalities","representation learning"],"falsifier":"A reader could rerun the five comparisons with identical hyperparameter budgets for every method and many seeds, then form paired differences; if the proposed model's edge over the strongest baseline falls inside the pooled run-to-run noise—and the paper's own reported standard deviations already show HGT's DOUBAN spread (0.1261) far exceeding its reported gain—the claim of consistent superiority fails.","tokens_in":15109,"feed_emoji":"🎯","tokens_out":10092,"duration_ms":90982,"temperature":0.7,"pith_summary":"The paper claims that multi-modal fusion should happen inside the propagation step of a graph neural network, not only before it or after it. In a multi-modal heterogeneous network, the similarity of two nodes on one modality should influence the weight of the message they exchange on another, because a pair of movies with very different texts but similar posters may still belong to the same genre. HGNN-IMA implements this with a nested inter-modal attention mechanism, a similarity-consistency modulation, and a loss that suppresses missing modalities, all inside the heterogeneous graph transformer architecture. The paper reports that this model outperforms every compared baseline on five real-world datasets, with the largest Macro-F1 gains on the two larger datasets. If the claim holds, practitioners should treat modality fusion as a propagation-time decision rather than an input or output blending step.","feed_headline":"Modalities steer each other's message passing for better node labels","feed_subtitle":"A nested attention mechanism fuses modalities during propagation, improving F1 on five real-world datasets.","key_machinery":"The Cross-modal Influence Unit is the central object. For each node pair $(v_i, v_j)$, it computes per-modality similarity scores $g^{(k),m'}_{ij}$ using type-dependent linear projections and edge-type-dependent matrices, converts them into inter-node attention $\\alpha^{(k),m'}_{ij}$, computes a second set of scores $s^{(k),m'}_{ij}$ with separate matrices $W^{\\text{MODAL}}$, softmaxes over modalities to get inter-modal attention $\\lambda^{(k),m'}_{ij}$, and forms the mixed attention $\\beta^{(k)}_{ij} = \\operatorname{softmax}_{j\\in\\mathcal{N}_i}\\left(\\sum_{m'} \\lambda^{(k),m'}_{ij} \\alpha^{(k),m'}_{ij}\\right)$. This nested attention lets one modality's similarity gate another modality's propagation while keeping the parameter count moderate. Two auxiliary mechanisms carry the robustness claims: alignment modulation $\\bar{\\beta}^{(k)}_{ij}$, which uses the absolute pairwise differences of per-modality similarity scores to down-weight misaligned neighbors, and the attention loss $\\mathcal{L}_{\\text{att}}$, which penalizes inter-modal attention assigned to modalities a neighbor does not actually possess.","core_discovery":"The central claim is that category-oriented representation learning on multi-modal heterogeneous networks requires modalities to influence each other during information propagation, not only before it (early fusion) or after it (late fusion). Concretely, when a node aggregates messages from a neighbor on one modality, the aggregation weight should be built from similarity scores on all modalities, with each modality's contribution itself weighted by an attention score that depends on the two nodes' features. Within the heterogeneous graph transformer framework, HGNN-IMA computes per-modality inter-node attention $\\alpha^{(k),m'}_{ij}$, learns a nested inter-modal attention $\\lambda^{(k),m'}_{ij}$ over modalities for each node pair, and combines them into a single propagation weight $\\beta^{(k)}_{ij}$. A modulation term based on cross-modality similarity consistency amplifies neighbors whose modalities agree, and an attention loss discourages high weights from missing modalities. The paper argues these components make learned embeddings align better with class labels, and supports this with consistent Macro-F1 improvements over all compared baselines on five datasets.","pith_inferences":["The nested-attention pattern is not tied to heterogeneous transformers; the same gating idea could be inserted into any attention-based message-passing model, so the paper's strongest legacy may be a reusable mechanism rather than a single architecture.","Because the inter-modal weights are computed for each node pair and layer, they provide a built-in diagnostic of which modality drives classification in which region of the graph, something the paper does not exploit.","A natural extension the paper does not run is a missing-modality robustness test: mask modalities at test time and measure whether the attention loss alone keeps performance from degrading.","The deliberate simplification that the mixed attention is independent of the influenced modality leaves a clear next question: does making it modality-specific help on tasks where one modality is far more informative than the others, at the cost of more parameters?"],"forward_implications":["Fusion during propagation outperforms both early fusion (HetGNN) and late fusion (MHGAT) on the five tested datasets, so the design point is worth adopting in networked multi-modal classification.","The three components are each load-bearing: removing the Cross-modal Influence Unit, the alignment modulation, or the missing-modality attention loss lowers Macro-F1 in the ablations.","The mixed inter-node attention $\\beta$ is more category-aligned than plain modality-specific attention: on AMAZON-1 it is larger for 54.5% of same-label pairs and smaller for 55.5% of different-label pairs.","The stated worst-case complexity of $O(|V|^2|M|^2)$, usually $O(|V| n |M|^2)$ for sparse neighborhoods, keeps the method scalable enough for the large datasets tested."],"supporting_citations":[{"why":"Supplies the heterogeneous graph transformer backbone that HGNN-IMA builds on, including type-dependent projection functions and edge-type-dependent matrices.","marker":"Hu et al., 2020"},{"why":"Contributes the MHGAT late-fusion baseline and the pre-encoded text/image features used for the DOUBAN, IMDB, and AMAZON datasets.","marker":"Jia et al., 2022"},{"why":"Provides the closest existing cross-modal propagation baseline (XGEA) that the paper contrasts with its adaptive nested inter-modal attention.","marker":"Xu et al., 2023"},{"why":"Defines the HetGNN early-fusion baseline that the paper's propagation-time fusion is designed to outperform.","marker":"Zhang et al., 2019"},{"why":"Defines the HAN heterogeneous graph attention baseline used in comparisons without special multi-modal treatment.","marker":"Wang et al., 2019"},{"why":"Supplies the CLIP encoders used to produce text and image features for the self-constructed AMAZON-1 and AMAZON-2 datasets.","marker":"Radford et al., 2021"},{"why":"Defines the IDKG baseline that treats the knowledge graph as an extra modality in the comparison.","marker":"Li et al., 2023a"},{"why":"Contributes the SeHGNN baseline, a strong typical heterogeneous GNN that the model must beat on several datasets.","marker":"Yang et al., 2023b"}],"fun_headline_variants":["Modalities co-steer propagation weights for sharper node labels","Nested inter-modal attention refines message passing","Fuse modalities during propagation, not just before or after","Mutual modality influence improves node classification in MMHNs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the baselines were tuned well enough and enough repeated runs were made that the reported accuracy differences, some as small as a few tenths of a percent, are not just run-to-run noise.","fun_headline_variants_meta":{"raw":{"variants":["Modalities co-steer propagation weights for sharper node labels","Nested inter-modal attention refines message passing","Fuse modalities during propagation, not just before or after","Mutual modality influence improves node classification in MMHNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2154,"prompt_tokens":976,"completion_tokens":1178,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1113}},"tokens_in":592,"tokens_out":1178,"duration_ms":11948,"temperature":1.0,"reasoning_tokens":1113,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:22:43.771036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun the five comparisons with identical hyperparameter budgets for every method and many seeds, then form paired differences; if the proposed model's edge over the strongest baseline falls inside the pooled run-to-run noise—and the paper's own reported standard deviations already show HGT's DOUBAN spread (0.1261) far exceeding its reported gain—the claim of consistent superiority fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the HetGNN early-fusion baseline that the paper's propagation-time fusion is designed to outperform."},{"cited_title":"Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro-video","cited_arxiv_id":null,"evidence_quote":"Defines the HAN heterogeneous graph attention baseline used in comparisons without special multi-modal treatment."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP encoders used to produce text and image features for the self-constructed AMAZON-1 and AMAZON-2 datasets."}],"review_version":1}