{"id":"049af1c7-3f78-45e1-b9ef-4c279fd5a574","arxiv_id":"2506.10282","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A unified comparison across six multimodal graph datasets shows that fine-tuned multimodal LLMs used as direct predictors achieve the highest node classification accuracy, even without graph structure input.","lead":"This paper presents Graph-MLLM, an open benchmark for multimodal graph learning that compares GNN-based, LLM-based, and multimodal-LLM-based methods across six datasets. Its main finding is that fine-tuning a multimodal LLM as a direct predictor, without explicit graph structure, often beats graph-aware baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finding 8 conflates paradigm quality with model capacity: the fine-tuned 7B predictor receives far more parameters and compute than the GraphLLM baselines, while Table 2 omits the closest competitor on Reddit due to time constraints.","rationale":"The reader's weakest_assumption identified the fairness of the cross-paradigm comparison, specifically missing compute budgets, seeds, and omitted Table 2 cells. My stress-test confirms this is the most load-bearing concern: the paper's headline finding is derived directly from cross-paradigm accuracy comparisons, and the resource asymmetry between a 7B fine-tuned MLLM and smaller frozen-encoder baselines is substantial. The omitted MLaGA/Reddit cell is particularly relevant because MLaGA is the strongest GraphLLM baseline and Reddit is one of the six evaluation datasets. My proposed test directly addresses whether the gap persists under matched capacity and budget. This does not invalidate the paper's plausible finding, but it supports the reader's CONDITIONAL verdict and its emphasis on statistical rigor and disclosed resource budgets.","tokens_in":14231,"tokens_out":4198,"duration_ms":55377,"concrete_test":"Run the strongest GraphLLM baseline (MLaGA, or LLaGA if MLaGA remains infeasible) with its LLM backbone fine-tuned using the same LoRA recipe, training budget, and GPU allocation as Qwen-VL-7B on all six datasets, with at least 5 seeds and reported mean±std. If MLaGA matches or surpasses Qwen-VL on any dataset where its Table 2 cell was omitted, Finding 8's 'even without explicit graph structure' conclusion is not robust to resource-controlled comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.5 / Finding 8 is that fine-tuned MLLM predictors achieve state-of-the-art results in most scenarios even without explicit graph structure information. The evidence for this is a comparison of best accuracies across Tables 1, 2, and 3, but the comparison is not resource-controlled. Qwen-VL-7B is a 7B-parameter model fine-tuned with LoRA on each dataset, while the GNN and GraphLLM baselines are evaluated with frozen CLIP embeddings and official implementations. No GPU-hours, trainable-parameter counts, seeds, or error bars are reported. Table 2 also marks several cells as omitted due to time-efficiency constraints, including MLaGA on Reddit and GraphTranslator on multiple datasets, whereas Table 3 reports Qwen-VL on all six datasets. The observed advantage of the MLLM predictor may therefore reflect model capacity and compute budget rather than the absence of graph structure. The result is plausible and worth testing, but as presented the benchmark does not isolate the paradigm from the resource axis it claims to compare.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Graph-MLLM, a benchmark for multimodal graph learning that organizes existing methods into three paradigms—MLLM-as-Encoder (GNN-based), MLLM-as-Aligner (GraphLLM-based), and MLLM-as-Predictor (MLLM-based)—and evaluates them on six node-classification datasets from Amazon and Reddit. The central empirical claims are that multimodal node attributes improve GNN performance, that MLLM-based image-to-text alignment has model- and dataset-specific effects, and that fine-tuned 7B MLLMs used directly as predictors achieve state-of-the-art accuracy on most datasets even without explicit graph structure. The benchmark code is open-sourced, and the taxonomy plus dataset coverage are useful contributions.","tokens_in":14583,"tokens_out":3075,"duration_ms":38300,"significance":"If the central claim holds, the paper would provide a valuable benchmark and a clear signal that MLLM-as-Predictor is a promising direction for multimodal graph learning. The open-source library, the six-domain evaluation, and the deliberate comparison of three paradigms are concrete strengths. However, the headline finding currently rests on cross-model comparisons that do not control for model capacity, compute, or tuning budget, and several baseline cells are missing. The result is plausible but not yet demonstrated at the level of the abstract's 'state-of-the-art in most scenarios' claim. The paper would be strengthened by resource-controlled comparisons, variance reporting, and completing the omitted baseline evaluations.","major_comments":[{"comment":"Finding 8, the paper's central claim, compares the best accuracies of fine-tuned 7B MLLMs against GNN and GraphLLM baselines without controlling for model capacity, parameter count, or compute budget. Qwen-VL-7B is a 7B-parameter model fine-tuned with LoRA, whereas the GNN baselines use frozen CLIP embeddings and the GraphLLM baselines are smaller, task-tuned models. No GPU-hours, trainable-parameter counts, seeds, or error bars are reported. Single-run differences as small as 0.01% (e.g., Table 3: 88.33 vs. 88.34 on Grocery) are treated as meaningful. As presented, the observed MLLM advantage may reflect scale rather than the paradigm, so the conclusion that fine-tuned MLLMs are state of the art 'even without explicit graph structure information' is not yet established. I would like to see either matched-capacity comparisons (e.g., larger GNN/GraphLLM backbones or a smaller MLLM) or, at minimum, a clear statement of compute budgets and variance across seeds.","section":"§4.5, Fig. 3; Tables 1–3"},{"comment":"Several high-performing baseline cells are missing due to 'time efficiency constraints,' including MLaGA on Reddit and GraphTranslator on Arts, CDs, and Reddit. MLaGA is identified in Finding 5 as the strongest GraphLLM, so omitting it on Reddit—one of the six datasets used in the aggregated comparison—removes the closest paradigm competitor from exactly the dataset where the fine-tuned MLLM predictor achieves its largest relative gains. The claim that fine-tuned MLLMs achieve state-of-the-art results in 'most scenarios' is therefore supported by an incomplete baseline matrix. The authors should either complete these runs or explicitly restrict the claim to the five datasets with complete MLaGA results.","section":"Table 2 and §4.5"},{"comment":"UniGraph2, the representative structural GNN foundation model, is evaluated without its shortest-path-distance (SPD) module, with the note that the original implementation is O(n^3). If SPD is a core component of UniGraph2's structural modeling, the benchmark understates the performance of the Encoder paradigm and weakens Finding 9, which attributes limited structure effectiveness to data properties rather than to the ablated baseline. The authors should either report UniGraph2 with the full model (or an efficient approximation) or explicitly state that the reported UniGraph2 numbers are an ablation and temper Finding 9 accordingly.","section":"Appendix A.2, Table 1b"},{"comment":"Finding 2 states that structure-aware alignment 'does not consistently enable' GNN-based models, yet the paper does not report statistical significance, and several differences in Table 1 are within a few tenths of a percent (e.g., 53.48 vs. 54.11 vs. 55.04 on CDs across MGAT variants). Without seeds, standard deviations, or significance tests, the distinction between 'consistent improvement' and 'no gain' is not reliable. This affects not only Finding 2 but also the paper's broader conclusion about when graph structure helps.","section":"§4.2, Findings 1–2; Table 1"}],"minor_comments":[{"comment":"The contribution bullet says the benchmark covers 'five popular multimodal graph datasets,' but the experiments use six datasets (Movies, Toys, Grocery, Arts, CDs, Reddit). Please correct the count.","section":"Abstract vs. Contributions (§1)"},{"comment":"There is a duplicated word: 'Notably, Notably, in the Reddit dataset.'","section":"§4.2, Finding 2"},{"comment":"The model name is inconsistently spelled as 'QWen-VL' (e.g., §3.2, §3.3, Appendix A.2) and 'Qwen-VL' (e.g., Table 3). Please use a single spelling.","section":"Throughout"},{"comment":"Figure 2 is described only as 'Different aligner augmentation methods comparison' without axis labels or a legend in the caption; the text refers to it but does not explain what is plotted. Please add a descriptive caption and define the compared settings.","section":"Figure 2 and §4.3"},{"comment":"The phrase 'consistent and stable improvements' is inaccurate when Qwen-VL-7B improves on only 3 out of 6 datasets with structure-aware fine-tuning. Please rephrase to convey the mixed results.","section":"§4.4, Finding 7"},{"comment":"The paper says it 'adheres to official guidelines for LoRA fine-tuning strictly' but does not report LoRA rank, learning rate, batch size, or training steps in the main text or appendix. Including these hyperparameters would improve reproducibility and help readers assess the resource-control issue in the major comments.","section":"§3.3 and Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is interesting, and the benchmark library is a useful contribution, but the current evidence does not yet support the abstract's strong statement. In addition to the technical revisions requested in the major comments, the editor may wish to confirm that the MLaGA baseline results are produced by the official implementation, since the authors of this paper also authored MLaGA; this is a disclosure concern rather than an accusation of bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on multimodal graphs: the paper delivers a systematic benchmark of three MLLM roles (Encoder, Aligner, Predictor) across six datasets, and it ships code. The taxonomy is clean, the datasets span e-commerce and social networks, and the finding that structure-aware prompting helps little once an MLLM is fine-tuned is a real, reproducible observation. The paper is also honest about several null results, which I appreciate.\n\nThe genuinely new part is the cross-paradigm comparison. Previous benchmarks (MM-Bench, MGB) only covered GNN encoders, so extending to GraphLLMs and direct MLLM predictors is a useful service. Findings 1–7 are mostly sensible. The problem is Finding 8, the headline: fine-tuned MLLMs achieve SOTA even without graph structure. That may be true, but the evidence is confounded. Qwen-VL-7B is a 7B-parameter model fine-tuned on each dataset, while the GNN and GraphLLM baselines are far smaller and evaluated with frozen CLIP embeddings. No GPU-hours, trainable-parameter counts, or seeds are reported. The comparison is best-accuracy against best-accuracy, so you cannot separate the paradigm from the resource axis. The internal comparison within Qwen (fine-tune with vs. without neighbor info) is controlled and suggests structure gives little, but the claim that this beats graph-structured baselines is not established.\n\nOther soft spots: Table 2 omits several cells, including MLaGA on Reddit and GraphTranslator on multiple datasets, citing time constraints; you can't know whether the omitted numbers would change the ranking. There are no error bars or seeds anywhere, which matters for accuracy differences of 1–2%. UniGraph2 is evaluated without its SPD module, and the authors don't quantify the impact. The self-citation to MLaGA is not hidden — it's reference [5] — but the text doesn't flag that it's their own model, which is a disclosure slip, not a scientific flaw.\n\nWho is this for? Researchers choosing approaches for multimodal graph tasks and anyone building on MM-Bench. The benchmark will be useful even if Finding 8 turns out to be a capacity artifact. It deserves serious peer review, but the revision should report compute budgets, error bars, and full tables, and soften the SOTA claim accordingly.","headline":"A genuinely useful three-paradigm benchmark for multimodal graph learning, but the headline claim that fine-tuned MLLMs beat graph-structured baselines is not yet supported because the comparison ignores model capacity and compute budgets.","tokens_in":15032,"tokens_out":1653,"would_cite":true,"duration_ms":21600,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned MLLMs beat graph models for multimodal node classification, even without graph structure.","keywords":["multimodal graph learning","multimodal large language models","benchmark","node classification","graph neural networks","graph LLMs","LoRA fine-tuning","image-to-text alignment"],"falsifier":"Run the same six datasets with per-method compute budgets (equal GPU-hours, multiple seeds, tuned hyperparameters) and report mean and standard deviation; if the best graph LLM or a structure-aware MLLM variant with a non-trivial graph encoder matches or exceeds the no-structure fine-tuned Qwen-VL accuracy, the central claim fails.","tokens_in":14016,"feed_emoji":"🤖","tokens_out":4620,"duration_ms":53071,"temperature":0.7,"pith_summary":"This paper benchmarks how multimodal large language models (MLLMs) can be used for learning on graphs whose nodes carry both text and images. It compares three paradigms—MLLM as feature encoder for graph neural networks, MLLM as aligner that converts images to text for graph-language models, and MLLM as a standalone fine-tuned predictor—across six product and social-network datasets. The central claim is that fine-tuning a general MLLM on node-level instructions yields the highest classification accuracy in most settings, and that supplying explicit graph structure gives at best marginal gains once the MLLM is fine-tuned. The paper also finds that combining text and image node attributes consistently helps GNN-based methods, and that converting images into textual descriptions helps some graph-language models but not others. It offers the benchmark as an open-sourced framework for fair, unified evaluation of the three paradigms.","feed_headline":"Fine-tuned MLLMs beat graph models without graph structure","feed_subtitle":"Six-dataset benchmark shows fine-tuned MLLMs win node classification even when edges are ignored.","key_machinery":"The organizing device is the three-way taxonomy of MLLM roles—Encoder, Aligner, and Predictor—which structures the comparison across six datasets. The predictor result is carried by instruction fine-tuning of MLLMs with LoRA on node-level prompts, while the encoder and aligner results rest on CLIP-based contrastive encoders (including a structure-aware variant CLIP-F-S) and on Qwen-VL-generated image summaries that are folded into textual prompts for graph LLMs.","core_discovery":"The paper establishes that fine-tuning a general MLLM (Qwen-VL-7B, LLaVA-1.5-7B) on each node's multimodal content, without feeding any adjacency or neighbor information, achieves state-of-the-art node classification accuracy on most of the six datasets, outperforming GNNs with CLIP encoders and graph-language models such as MLaGA and UniGraph2. Injecting neighbor text or images into the MLLM prompt produces improvements of less than one percentage point in most cases, and including neighbor images often hurts. This MLLM-as-Predictor strategy is identified as the most promising of the three paradigms for multimodal graph learning, with the caveat that the value of structure awareness depends heavily on graph density and image quality.","pith_inferences":["If the central claim holds, architectural research on graph-specific models may be less urgent for node classification on attribute-rich multimodal graphs than improving instruction-tuning data and MLLM efficiency; the graph's value could lie mainly in providing more node-level supervision rather than topology.","A natural extension is to test MLLM predictors on graphs where node attributes are missing or very weak—there structure may become indispensable, and the paper's ranking would likely reverse.","The consistent degradation when neighbor images are added to prompts suggests MLLMs struggle to process multiple images coherently, so a learned graph tokenizer rather than prompt stuffing might recover the structure gains the paper finds marginal.","A fair-compute re-run that reports per-method GPU budgets, seeds, and error bars could change the headline ranking; that caution is our inference, not the paper's claim."],"forward_implications":["MLLM-as-Predictor should become the default baseline and likely backbone for multimodal graph node classification.","Explicit graph structure is not necessary for strong performance when node text and images are rich; sparse or noisy structure can be ignored, and injecting it may degrade accuracy.","Converting node images to textual descriptions is a viable way to bring visual information into text-only graph LLMs, but its benefit depends on both the underlying model architecture and the dataset.","Structure-aware methods help mainly on dense graphs with high-quality images (such as the CDs dataset) and fail on sparse graphs or when images are weak.","Fine-tuned MLLMs without structure can match or exceed dedicated multimodal graph foundation models, implying the graph-specific inductive bias is not essential for this task class."],"supporting_citations":[{"why":"Supplies Qwen-VL-7B, the MLLM used as the fine-tuned predictor and the image-to-text aligner; the predictor results depend on its LoRA fine-tuning behavior.","marker":"[2]"},{"why":"Supplies LLaVA-1.5-7B, the second MLLM predictor whose fine-tuning confirms the MLLM-as-Predictor finding across datasets.","marker":"[18]"},{"why":"MLaGA is the state-of-the-art graph-LLM baseline that fine-tuned MLLM predictors are claimed to match or beat without structure information.","marker":"[5]"},{"why":"UniGraph2 is the state-of-the-art multimodal GNN foundation model baseline against which fine-tuned MLLMs and aligned graph LLMs are compared.","marker":"[11]"},{"why":"CLIP supplies the pre-trained multimodal encoder for GNN features and GraphLLM embeddings; the Encoder-paradigm results and several baselines rest on it.","marker":"[22]"},{"why":"Provides the evaluation protocol and dataset preprocessing the paper follows to ensure a common experimental setup across paradigms.","marker":"[36]"}],"fun_headline_variants":["Fine-tuned MLLMs beat graph models even without edges","Graph-less MLLMs outperform GNNs on node classification","MLLMs ignore edges, still top graph benchmarks","No graph structure needed: Fine-tuned MLLMs win node tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison across paradigms is fair: all methods receive comparable compute, training budget, and hyperparameter tuning, with no seeds or error bars reported and some baseline cells omitted for time efficiency.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned MLLMs beat graph models even without edges","Graph-less MLLMs outperform GNNs on node classification","MLLMs ignore edges, still top graph benchmarks","No graph structure needed: Fine-tuned MLLMs win node tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2723,"prompt_tokens":1025,"completion_tokens":1698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1627}},"tokens_in":641,"tokens_out":1698,"duration_ms":12120,"temperature":1.0,"reasoning_tokens":1627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:29:41.113059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six datasets with per-method compute budgets (equal GPU-hours, multiple seeds, tuned hyperparameters) and report mean and standard deviation; if the best graph LLM or a structure-aware MLLM variant with a non-trivial graph encoder matches or exceeds the no-structure fine-tuned Qwen-VL accuracy, the central claim fails.","supporting_citations":[{"cited_title":"Unigraph2: Learning a unified embedding space to bind multimodal graphs","cited_arxiv_id":null,"evidence_quote":"UniGraph2 is the state-of-the-art multimodal GNN foundation model baseline against which fine-tuned MLLMs and aligned graph LLMs are compared."}],"review_version":1}