REVIEW 4 major objections 6 minor 1 cited by
Vision Graph Prompting via Semantic Low-Rank Decomposition
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Semantically connected vision-graph nodes share dominant PCA components, so a frozen ViG with three low-rank prompts matches full fine-tuning.
desk verdict A useful PEFT method for Vision GNNs, but the low-rank semantic story is likely a KNN artifact and the Eq. 9/12 mismatch needs fixing before reproduction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the semantic low-rank decomposition: a shared MLP $MLP_s: \mathbb{R}^d \to \mathbb{R}^r$ maps each $d$-dimensional node feature to a compact semantic vector with $r \ll d$, and three low-rank prompt matrices $P_g, P_e, P_n \in \mathbb{R}^{r \times d}$ fold that vector back into the full feature space at the graph, edge, and node levels. SeLo-Graph uses virtual nodes to alter topology, while SeLo-Edge and SeLo-Node modify the aggregation and update steps of graph convolution respectively. This carries the argument by ensuring the trainable prompt capacity is spent on the presumed semantic subspace, filtering high-frequency local detail.
What would settle it
Take a ViG and compute the PCA spectrum of node features for three graph constructions — KNN, spatial adjacency, and random neighborhoods; if the eigenvalue decay is just as sharp for non-semantic neighborhoods, the low-rank 'semantic' claim is an artifact, and VGP should be re-tested with random low-rank prompt projections as a control.
Extended reading notes
Core claim
The paper's central claim is that semantically connected nodes in a ViG graph share dominant PCA components, so semantic information lies in a low-dimensional subspace of the latent feature space. To exploit this, VGP projects each node feature into a low-rank semantic space with a shared MLP and composes it with three prompt matrices: SeLo-Graph appends trainable virtual nodes that dynamically link to nearby nodes, SeLo-Edge propagates the aggregated low-rank neighbor semantics into each node update, and SeLo-Node blends the low-rank semantic component back into the original node feature. The appendix estimates the semantic rank at roughly 50 to 60 against a feature dimension of 768. With the backbone frozen, only the prompts and a downstream head are trained; experiments report an average accuracy of 89.6% versus 89.0% for full fine-tuning on the ten vision datasets, and gains on seven of nine chemistry and biology graph datasets.
Load-bearing premise
The load-bearing assumption is that semantically connected graph nodes share dominant PCA components because visual semantics is genuinely low-rank; since edges are drawn by K-nearest-neighbor search in the same feature space, this concentration could be an artifact of the graph-construction rule rather than a fact about semantics.
Editorial extensions
If this is right
- A frozen ViG can be adapted to downstream classification at accuracy comparable to full fine-tuning, cutting trainable parameters by about 94.6% on average and adding only about 3.1% extra FLOPs.
- On the ten vision benchmarks, VGP beats the strongest applied visual-prompting baseline by an average of 5.0 accuracy points, including on fine-grained and small-sample datasets.
- The same three prompt modules transfer to chemistry and biology graph classification, outperforming full fine-tuning on seven of nine datasets and previous graph-prompting adapters.
- The method is robust to its main hyperparameters: rank $r=32$ and blending factors $\alpha=\beta=0.2$ work across datasets with a broad plateau.
Reading between the lines
- An extension the paper does not test: because ViG builds edges by K-nearest-neighbor search in the very feature space used for the PCA analysis, the low-rank 'semantic' property may be partly a by-product of that construction rule; re-running the prompts on spatially built graphs would separate semantic low-rankness from KNN-induced similarity.
- If the low-rank semantics is generic across graph backbones, the same three prompts could be applied to other graph-based vision models or point-cloud GNNs by retraining only the low-rank MLP and prompt matrices, without changing the recipe.
- The reported sensitivity table suggests rank could be set per block from the PCA spectrum of that block's features, rather than fixed globally at 32, potentially improving parameter efficiency on deeper models.
- The strong graph-domain results imply a testable hypothesis the paper leaves open: molecular and protein graphs may inherit low-rank structure from their pre-training features, so VGP-style prompts might work even without task-specific graph pre-training.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Vision Graph Prompting (VGP), a parameter-efficient fine-tuning method for Vision GNNs. It introduces three prompt components—SeLo-Graph, SeLo-Edge, and SeLo-Node—that are designed around the claimed observation that semantically connected nodes in vision graphs share low-rank PCA structure. The method is evaluated on ten image classification datasets and nine chemistry/biology graph benchmarks, reporting accuracy comparable to or better than full fine-tuning while updating roughly 5% of the parameters. The paper also includes ablations of the three components, rank dimension, and blending factors, plus an efficiency analysis.
Significance. If the claims hold, VGP would be a useful contribution to parameter-efficient fine-tuning for graph-based vision backbones, an area with few existing prompting methods. The paper's strengths are the breadth of evaluation (external benchmarks for both vision and graph tasks), the component-level ablation, the efficiency table, and the public code release. However, the central 'low-rank semantic' insight is not established as a property of semantics rather than a byproduct of the KNN graph construction, and the headline comparison with full fine-tuning lacks statistical support. The method may still work as an adapter, but the paper as written overstates the interpretability and the quantitative certainty of its core claims.
major comments (4)
- [Section 4.3, Eq. (9) and Section 4.5, Eq. (12)] The two formal specifications of the edge prompt are not algebraically equivalent. Equation (9) applies a normalized convex combination to the center node before convolution, while Eq. (12) applies the edge term as an unnormalized additive residual after the update and drops the normalization. The manuscript provides no derivation linking the two, and the ablation in Table 3 does not state which formulation was implemented. Because Tables 1 and 3 depend on the exact edge-prompt update, a reader cannot reproduce the reported numbers from the text; please specify the implemented form, justify it, and reconcile Eq. (9) with Eq. (12).
- [Section 5.3, Tables 1-3] No standard deviations, number of seeds, or significance tests are reported. The headline claim 'comparable to full fine-tuning' rests on average accuracy gaps (e.g., 89.6 vs. 89.0 in Table 1) that are smaller than typical run-to-run variation on fine-grained benchmarks, and several per-dataset entries (CUB, NABirds, Flowers, CIFAR10) are actually lower than full fine-tuning. Please report multiple seeds with error bars and a statistical test for the key comparisons, especially the comparison against full fine-tuning and against the strongest baseline InsVP.
- [Section 3 and Appendix A.3] The observation that connected nodes share dominant PCA components is confounded by the KNN rule in Eq. (1). Since ViG connects each node to its nearest neighbors in the same feature space, connected nodes are close by construction and will share principal components even without semantic structure. The visualizations in Figure 2 and Appendix A.3 do not include a control condition (e.g., random node pairs matched for feature distance, or KNN graphs on non-semantic features), so the paper's stated core insight that 'semantic information within vision graphs predominantly resides in the low-rank components' is not currently supported. Please add such a control or temper the claim accordingly.
- [Section 5.3, Table 2] The extension to chemistry/biology tasks is justified by the hypothesis that low-rank patterns exist in molecular and protein graphs, but no analysis of that property is provided for these datasets. Since the entire prompt design is motivated by the low-rank semantic observation, the graph-task results do not demonstrate that the low-rank mechanism transfers; they only show that the specific set of prompts improves GIN prompting baselines. Please provide a low-rank analysis on graph datasets or reframe the contribution of these experiments.
minor comments (6)
- [Section 4.2, Eq. (6)] Equation (6) writes cosine similarity as a raw dot product, but cosine similarity normally requires normalization; please clarify whether the top-K selection uses normalized features.
- [Section 4.3, Eq. (9)] Equation (9) uses \|N_s(x_c)\| to denote set cardinality, which is unconventional, and the parenthesis after N_s(x_c) is unclosed. Please fix the notation.
- [Section 5.4, Figure 4] Figure 4 has no axis labels and the red star is not described in the caption, so the dependence of accuracy on alpha and beta cannot be read reliably. Please add axes labels, a legend, and error bars if multiple seeds were used.
- [Appendix A.3] Appendix A.3 estimates an observed rank of 50-60 for CUB and Flowers, but the experiments use r=32; the link between the observed rank and the chosen hyperparameter is not explained. Please clarify how the chosen rank follows from the analysis.
- [Section 5.2] The statement that all methods were trained for 100 epochs and then '10 additional epochs for probing the optimal result' is ambiguous; please clarify whether this is an evaluation protocol, early stopping, or a model selection procedure.
- [Appendix A.1, Table 5] Table 5 reports FLOPs only for full fine-tuning and VGP; without the same metric for the other PEFT baselines, the conclusion of 'little additional overhead' is not fully contextualized. Please include FLOPs for the compared methods or qualify the claim.
Circularity Check
No significant circularity: reported results come from external benchmarks, not from the low-rank observation.
full rationale
The paper's derivation chain is: ViG constructs a KNN graph in feature space (Eq. 1); PCA on those features shows connected nodes share dominant components; prompts are therefore designed as low-rank decompositions (Eqs. 5, 8-10); and the resulting frozen-backbone adapter is evaluated on ten vision and nine graph benchmarks. None of the reported accuracies in Tables 1-3 is produced by substituting the low-rank observation into an equation; the observation is used only as design motivation. The evaluation is external, with standard splits, and the method's parameters are optimized on the training splits, which is ordinary fine-tuning, not circularity. The only element with a flavor of triviality is the observation itself: since ViG selects neighbors by K-nearest-neighbor in the same feature space, connected nodes are close in that space, so their dominant PCA components will tend to agree by construction. That makes the 'semantic low-rank' insight partially a restatement of the KNN rule rather than an independent empirical discovery, but it does not infect the benchmark results. No load-bearing self-citation is present: the authors' prior InsVP appears only as a baseline. No uniqueness theorem is imported. The paper's explicit 'we hypothesize' for chemistry/biology low-rank structure (Section 5.3) is a stated hypothesis, not a derived prediction. The inconsistency between Eq. 9 and Eq. 12 is a reproducibility gap, not circularity. Overall score 0.
Assumptions & free parameters
free parameters (5)
- Rank r =
32
- Blending factor alpha =
0.2
- Blending factor beta =
0.2
- Number of graph prompt nodes M =
not reported
- PCA threshold epsilon =
0.25
assumptions (4)
- domain assumption K-nearest-neighbor graph construction in feature space carries semantic relationships (ViG premise).
- domain assumption Semantic content of an image graph is concentrated in a low-rank subspace of features.
- domain assumption The same low-rank assumption transfers to molecular and protein graph datasets.
- domain assumption Freezing all backbone parameters still leaves enough representation for transfer.
Cite this review
Pith. "Pith review of Vision Graph Prompting via Semantic Low-Rank Decomposition." pith.science (2026). https://pith.science/paper/JURU5HCY
@misc{pith2026250504121,
author = {Pith},
title = {Pith review of: Vision Graph Prompting via Semantic Low-Rank Decomposition},
year = {2026},
howpublished = {\url{https://pith.science/paper/JURU5HCY}},
note = {Machine review of arXiv:2505.04121}
}
read the original abstract
Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditional grid or sequence-based representations. To efficiently adapt ViG to downstream tasks, parameter-efficient fine-tuning techniques like visual prompting become increasingly essential. However, existing prompting methods are primarily designed for Transformer-based models, neglecting the rich topological relationships among nodes and edges in graph-based representations, limiting their capacity to model complex semantics. In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures. Our core insight reveals that semantically connected components in the graph exhibit low-rank properties. Building on this observation, we introduce a semantic low-rank prompting method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-grained semantic dependencies. Extensive experiments demonstrate our method significantly improves ViG's transfer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our code is available at https://github.com/zhoujiahuan1991/ICML2025-VGP.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
UPP: Unified Point-Level Prompting for Robust Point Cloud Analysis
UPP unifies denoising and completion as point-level prompts in a frozen pre-trained point cloud transformer, improving classification accuracy on noisy and incomplete point clouds without updating backbone weights.
Reference graph
Works this paper leans on
-
[1]
Exploring visual prompts for adapting large-scale models
Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274,
-
[4]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,
2009
-
[5]
Attributed to our semantic low-rank decomposition design, we save a lot of computation by reducing feature dimension to low-rank dimension for computing, while extracting critical semantic information and filtering out disruption of noisy local details. Our approach does not require the re-training of the entire model, which helps to mitigate the computat...
work page 2017
-
[8]
Vi- sion hgnn: An image is more than a graph of nodes
Han, Y ., Wang, P., Kundu, S., Ding, Y ., and Wang, Z. Vi- sion hgnn: An image is more than a graph of nodes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19878–19888, 2023b. Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V ., and Leskovec, J. Strategies for pre-training graph neural networks. arXiv preprint a...
arXiv 1905
-
[13]
Y ., et al
Netzer, Y ., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y ., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp
2011
-
[14]
and Zisserman, A
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722–729. IEEE,
2008
-
[2008]
Schick, T. and Sch¨utze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676,
arXiv 2001
-
[2009]
An image is worth 16x16 words: Trans- formers for image recognition at scale
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929,
arXiv 2010
Show all 18 references
-
[2011]
Kipf, T. N. and Welling, M. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016a. Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016b. Krizhevsky, A., Hinton, G., et al. Learn...
-
[2014]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,
1901
-
[2015]
The caltech-ucsd birds-200-2011 dataset,
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset,
2011
-
[2017]
The learning rate is set as 0.001 and the weight decay is 0.05
optimizer for optimization and implement cosine learning rate annealing. The learning rate is set as 0.001 and the weight decay is 0.05. Regarding the graph tasks, we follow the approach of GPF-Plus (Fang et al., 2023), utilizing a widely used 5-layer GIN as the underlying arc...
2023
-
[2020]
M., Yang, Y ., and Wang, C
Fang, T., Zhang, Y . M., Yang, Y ., and Wang, C. Prompt tuning for graph neural networks. arXiv preprint arXiv:2209.15240,
-
[2021]
Graphprompt: Uni- fying pre-training and downstream tasks for graph neural networks
Liu, Z., Yu, X., Fang, Y ., and Zhang, X. Graphprompt: Uni- fying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, pp. 417–428,
2023
-
[2022]
Food-101– mining discriminative components with random forests
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European con- ference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pp. 446–461. Springer,
2014
-
[2023]
Eˆ 2vpt: An effective and efficient approach for visual prompt tuning
Han, C., Wang, Q., Cui, Y ., Cao, Z., Wang, W., Qi, S., and Liu, D. Eˆ 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023a. Han, K., Wang, Y ., Guo, J., Tang, Y ., and Wu, E. Vision gnn: An image is worth graph of nodes. A...
-
[2024]
Stop: Integrated spatial-temporal dynamic prompting for video understanding
Liu, Z., Xu, K., Su, B., Zou, X., Peng, Y ., and Zhou, J. Stop: Integrated spatial-temporal dynamic prompting for video understanding. arXiv preprint arXiv:2503.15973,
-
[2025]
and Hutter, F
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05101,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.