REVIEW 5 major objections 6 minor 16 references
SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read An unsupervised graph autoencoder can rank images by semantic content, beating caption-supervised graph models and vision models.
desk verdict SCENIR is a sensible unsupervised GAE for scene graph retrieval, but its GED evaluation metric shares the same SBERT embedding space as the model's input features, so the headline performance lead is not yet convincing evidence of semantic retrieval. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the SCENIR graph autoencoder: a variational graph encoder with two independent three-layer GNN branches (one for mean, one for variance), dual MLP decoders that reconstruct the adjacency matrix and the node feature matrix, and an adversarial discriminator that regularizes the latent space toward a Gaussian prior. At inference the mean branch's node embeddings are sum-pooled to a graph vector, and retrieval is cosine ranking. Graph Edit Distance, with node substitution costs defined as cosine distance over 768-dimensional class embeddings, serves as the deterministic evaluation oracle that all reported metrics are computed against.
What would settle it
Take a random sample of PSG image pairs, have humans rate semantic similarity, and compare those ratings with the approximate GED rankings used here; low rank correlation would show the evaluation oracle, and with it the central claim, does not capture human semantics. A cheaper partial falsifier is to compute exact GED on small graphs and measure how often the approximation used in the paper changes the top-k ranking.
Extended reading notes
Core claim
The central claim is that a fully unsupervised graph autoencoder can learn scene graph embeddings that support more semantically faithful image retrieval than supervised alternatives, provided the architecture is adjusted to scene graphs. SCENIR splits the variational encoder into independent mean and variance GNN branches, decodes with two parallel 2-layer MLPs that reconstruct both adjacency and node features, and adds adversarial regularization; at inference it sum-pools the mean latent node embeddings and ranks candidates by cosine similarity. On the PSG dataset, the best configuration (with a GIN backbone) reaches NDCG@1 of 31.39 versus 29.64 for the best supervised IRSGS variant, and the unsupervised model remains the top ranker on in-the-wild Flickr30K images whose scene graphs are generated automatically. The paper also asserts that Graph Edit Distance, approximated with embedding-based node substitution costs, gives a deterministic ground truth for evaluating scene graph retrieval, replacing the caption-based supervision that shifts with the choice of sentence encoder.
Load-bearing premise
The ranking that all results depend on treats approximate Graph Edit Distance, with node costs derived from Sentence-BERT class embeddings, as the true measure of semantic similarity between images; if that measure does not track what people mean by semantic similarity, the reported superiority is superiority at this metric.
Editorial extensions
If this is right
- Image retrieval can be built without caption or similarity labels, removing the dependency on text encoders whose disagreements propagate into trained retrievers.
- Since preprocessing, training, and inference are all linear in dataset size, the approach scales to graph corpora where quadratic label construction would be prohibitive.
- Using one deterministic graph-distance ground truth makes retrieval results reproducible and comparable across research groups, unlike caption-based rankings.
- The same embeddings support downstream uses such as counterfactual image explanations, where SCENIR outperforms a supervised counterfactual framework in an inductive setting.
Reading between the lines
- Extension: if GED with cosine node costs is replaced by human-annotated semantic similarity, the relative ranking of methods is an open empirical question; the paper's own strongest claim is specifically about matching its GED-derived rankings.
- The success of a simple 3-layer GIN encoder plus MLP decoders suggests that over-smoothing, not depth, is the binding constraint for scene graphs, so deeper architectures may pay off if skip connections or normalizers are added.
- Because retrieval happens in embedding space, the same SCENIR encoder could be plugged into text-to-image or cross-modal pipelines by aligning scene graph embeddings with text embeddings, a direction the paper does not pursue.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SCENIR, an unsupervised graph autoencoder for image-to-image retrieval via scene graphs. SCENIR encodes scene graphs with split GNN encoder branches, reconstructs node features and adjacency with MLP decoders, and applies adversarial regularization; retrieval is performed by cosine similarity on sum-pooled latent embeddings. The authors evaluate on PSG against vision, vision-language, supervised GNN (IRSGS), and GAE baselines, using approximate Graph Edit Distance over 1K test graphs as ground truth. They report superior retrieval metrics for SCENIR-GIN, linear preprocessing/training/inference complexity, and additional experiments on Flickr30K and counterfactual retrieval. The GitHub repository containing the source code is provided.
Significance. If the results hold, the paper makes a useful contribution: it demonstrates that unsupervised graph autoencoders can be competitive with or superior to supervised scene-graph retrieval, and it advocates a deterministic GED-based evaluation protocol. The efficiency gain (8 minutes versus 50 minutes for IRSGS, and versus 3 hours for GC) and the inductive counterfactual application are concrete strengths. However, the central empirical claim depends on the validity of the GED ground truth and on the statistical robustness of the comparisons; both require additional support before the significance can be fully assessed.
major comments (5)
- [3.1, Appendix A and B] The evaluation metric is built from the same Sentence-BERT class embeddings that serve as the model's input node features. Appendix A states that X uses 768-dimensional Sentence-Transformer embeddings of the 189 object and predicate classes, and Appendix B defines the GED node substitution cost as the cosine distance between exactly those embeddings. Because SCENIR's loss includes feature reconstruction (Eq. 3, L_featrecon), a model that preserves its input features will by construction rank well under this GED. This makes every reported metric in Tables 1-4 partially endogenous to the model's input representation, so the results may reflect alignment with an SBERT-derived similarity rather than semantic relevance. To support the semantic-retrieval claim, the authors should validate the GED ranking against human judgments, test stability across multiple Sentence-Transformer models, and compare against exact GED or a well-established approximation.
- [4.1, Table 1] The strongest quantitative claims are not supported by the reported numbers. In Table 1, SCENIR-GIN has MRR 59.01, while IRSGS-GIN with RoBERTa supervision has MRR 59.16, so SCENIR is not superior on MRR; NDCG@10 is essentially tied (26.28 vs. 26.00). The text in Section 4.1 claims SCENIR "surpasses even the supervised IRSGS across most metrics," but the abstract and conclusion claim "superior performance across metrics," which is contradicted by Table 1. Since the paper reports no error bars, multiple seeds, or significance tests, even the clearer gaps (e.g., NDCG@1 31.39 vs. 29.64) cannot be distinguished from run-to-run variation. The authors should report means and standard deviations over several runs, perform significance tests, and adjust the strength of their claims accordingly.
- [Appendix B] The description of the GED approximation is incomplete and does not support the paper's claim that GED is a "deterministic and robust ground truth." Exact GED is NP-hard (as cited in Section 2), yet the authors compute pairwise GED scores for 1,000 test graphs, yielding roughly half a million comparisons; they do not specify the approximation algorithm, the edge edit costs, or the accuracy of the approximation relative to exact GED. Without this information, the reader cannot assess whether the ground-truth rankings are reliable or how sensitive they are to the approximation's parameters. The authors should provide the algorithm, its parameters, and a validation study on a sample of graphs.
- [4 and Appendix D] No validation split is described. The loss trade-off weights (lambda1=3, lambda2=1/6, lambda3=1/3) and the 3-layer encoder are selected through performance sweeps reported in Appendix D (Figures 8 and 9). If these sweeps were performed on the same 1K test graphs used to produce Tables 1-4, the reported results are optimistically biased and the comparisons are not statistically valid. The authors must clarify whether model selection used a separate validation set and, if so, provide its size and construction; otherwise, the hyperparameter tuning is effectively performed on the test set.
- [4.1, Baselines] The comparison with IRSGS is asymmetric because IRSGS is trained on caption-based SBERT similarity labels while the evaluation uses GED. This is a meaningful design choice, since the paper argues that caption supervision is inconsistent, but it means Table 1 does not establish that SCENIR learns GED better; it establishes that caption-trained models transfer worse to a GED-based metric. To make the comparison more conclusive, the authors should either include an IRSGS variant trained with GED-based labels, report the caption-based evaluation as well, or explicitly frame the result as a transfer comparison between training signals rather than as a direct superiority claim.
minor comments (6)
- [4.2, Figure 6 caption] DeiT is a vision transformer, not a vision-language model; the caption "VL (DeiT)" and the text "In Figure 6, DeiT, a VL model" are incorrect and should say "Vision."
- [4.4] There is a typo: "SCENIR achievessignificant" should be "SCENIR achieves significant." Also, "SotA" is used as an adjective inconsistently throughout the paper; consider spelling out "state-of-the-art" on first use.
- [Appendix E] The first sentence references "the qualitative results of Figures 5 and 6 of the main paper," but the qualitative results appear in Figures 6 and 7; the cross-reference should be corrected.
- [Main text Figure 5 and Appendix D Figure 8] The main-text Figure 5 is labeled as NDCG@5 versus number of GNN layers, while Appendix D Figure 8 reports NDCG@3, MAP@3, and MRR for the same layer sweep; this inconsistency could confuse readers and should be reconciled.
- [Appendix A] The Sentence-Transformer model used to construct the 768-dimensional class embeddings is not specified (no model name or version). Since the paper criticizes SBERT variability, the evaluation's dependence on one particular embedding model should be stated explicitly and justified.
- [Introduction references] The citation "Puaduraru et al., 2024" appears to contain a misspelling (likely "Pudararu"); please verify the reference.
Circularity Check
Evaluation ground truth is built from the same Sentence-Transformer class embeddings used as node features, so SCENIR's reported semantic lead is partly metric alignment; the GED framework is adopted from same-author prior work without external validation.
-
self definitional
[Appendix A (Dataset Preprocessing) and Appendix B (Ground Truth and Retrieval Metrics); training loss in Eq. (3)]
"We used 768-dimensional Sentence-Transformer embeddings for the 189 object and predicate classes to construct the feature matrix X for each scene graph, which is required as input for all the GNN models. ... we define the node substitution cost as the cosine distance between the node embeddings, while the node insertion/deletion cost is defined as the cosine distance to the mean node embedding (average of all the 133 object class embeddings)."
The GED ground-truth ranking is defined by cosine distances among Sentence-Transformer class embeddings, while exactly the same 768-dimensional Sentence-Transformer embeddings are used as the node feature matrix X consumed by every GNN, including SCENIR. SCENIR is trained to reconstruct X (L_featrecon = MSE(X, Z_f) in Eq. 3), so the evaluation target is a function of the same representation the model is trained to preserve. A model that reconstructs X well will, by construction, score well against this GED. The paper's claim that GED captures 'important semantic information' therefore circularly validates the model against its own input embedding geometry rather than an independently established semantic ground truth.
-
self citation load bearing
[Section 3.1 and Section 4, 'Ground Truth and Evaluation']
"We employ approximate GED as the ground truth distance/similarity for evaluating our approach, motivated by recent work adjacent to our field (Dimitriou et al., 2024) that emphasizes semantic similarity over low-level features, such as pixels. In accordance to their analysis and our experimental findings, GED’s robustness eliminates ambiguity in generating golden rankings, unlike methods such as captioning (Fig. 2)."
The cited Dimitriou et al. (2024) work shares three authors with the present paper (Dimitriou, Lymperaiou, and Stamou). The paper's central evaluation contribution—that approximate GED is a 'deterministic and robust ground truth measure'—is justified by 'their analysis' in this same-author prior work. Appendix B specifies only substitution and insertion/deletion cost functions, not the GED approximation algorithm, and no comparison to exact GED or to human semantic relevance is provided. The load-bearing premise that GED is a reliable semantic ground truth therefore rests on a self-citation rather than on independent verification.
full rationale
The derivation of SCENIR's embeddings is not circular in the narrow sense: the autoencoder is trained with an unsupervised reconstruction loss on X and A, and GED scores are never used as training labels. There is genuine independent content in the architectural study and in the runtime comparison. However, the semantic-evaluation claim is substantially endogenous. Appendix A builds the node feature matrix X from Sentence-Transformer class embeddings; Appendix B builds the GED ground truth from cosine distances over the same class embeddings. Because SCENIR is trained to reconstruct X, its latent space is biased toward preserving exactly the geometry in which the evaluation metric is defined, which can explain part of its advantage over caption-supervised IRSGS without implying that the metric tracks human-perceived semantics. The paper criticizes SBERT caption variability but does not test the stability of the class-name Sentence-Transformer embeddings used in the GED costs, and it imports the GED evaluation framework from the authors' own prior work. The structural component of GED, involving edge edit operations, is not fully specified and may add some independent signal, so this is partial rather than total circularity. Overall, the central claim still has independent content, but the evaluation metric is not an independent semantic benchmark, yielding a score of 4.
Assumptions & free parameters
free parameters (4)
- Loss tradeoff weights lambda_1, lambda_2, lambda_3 =
3, 1/6, 1/3
- Number of GNN encoder layers =
3
- GED edit costs =
cosine distance between SBERT class embeddings; cosine distance to mean class embedding
- Latent dimension and decoder widths =
1000 latent, 32 edge decoder, 768 feature decoder
assumptions (4)
- domain assumption Approximate GED with cosine-distance edit costs is a faithful proxy for semantic image similarity.
- domain assumption PSG scene graphs and Sentence-BERT class embeddings provide a semantically valid representation of image content.
- domain assumption Unsupervised graph reconstruction produces graph-level embeddings that preserve the semantic ordering induced by GED.
- domain assumption Scene graph generation (PSGTR) and captioning (BLIP) produce reliable annotations for Flickr30K evaluation.
Cite this review
Pith. "Pith review of SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval." pith.science (2026). https://pith.science/paper/YBL6RKKT
@misc{pith2026250515867,
author = {Pith},
title = {Pith review of: SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/YBL6RKKT}},
note = {Machine review of arXiv:2505.15867}
}
read the original abstract
Despite the dominance of convolutional and transformer-based architectures in image-to-image retrieval, these models are prone to biases arising from low-level visual features, such as color. Recognizing the lack of semantic understanding as a key limitation, we propose a novel scene graph-based retrieval framework that emphasizes semantic content over superficial image characteristics. Prior approaches to scene graph retrieval predominantly rely on supervised Graph Neural Networks (GNNs), which require ground truth graph pairs driven from image captions. However, the inconsistency of caption-based supervision stemming from variable text encodings undermine retrieval reliability. To address these, we present SCENIR, a Graph Autoencoder-based unsupervised retrieval framework, which eliminates the dependence on labeled training data. Our model demonstrates superior performance across metrics and runtime efficiency, outperforming existing vision-based, multimodal, and supervised GNN approaches. We further advocate for Graph Edit Distance (GED) as a deterministic and robust ground truth measure for scene graph similarity, replacing the inconsistent caption-based alternatives for the first time in image-to-image retrieval evaluation. Finally, we validate the generalizability of our method by applying it to unannotated datasets via automated scene graph generation, while substantially contributing in advancing state-of-the-art in counterfactual image retrieval.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[3]
Results for NDCG@3, MAP@3 and MRR of the SCENIR-GIN model, reported in Figure 9. Figure 10. Left: illustration of the encoder architecture in original GAE (unified), and in the proposed SCENIR (split). Right: Performance comparison between split and unified architecture, on MAP@3, MRR and NDCG@3. Finally, regarding the splitting of the GNN Encoder, the or...
work page 2024
-
[8]
URL https://api.semanticscholar. org/CorpusID:257039217. Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., and Zhang, C. Adversarially regularized graph autoencoder for graph embedding, 2018. Park, N. and Kim, S. How do vision transformers work? 2022. Peng, Y . and Chi, J. Unsupervised cross-media retrieval using domain adaptation with scene graph. IEEE Tran...
work page 2018
-
[9]
URL https://api.semanticscholar. org/CorpusID:273549566. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using ...
work page 2021
-
[10]
URL https://api.semanticscholar. org/CorpusID:201646309. Sanfeliu, A. and Fu, K.-S. A distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13 (3):353–362, 1983. doi: 10.1109/TSMC.1983.6313167. 10 SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval Shi,...
arXiv 1983
-
[11]
URL https://api.semanticscholar. org/CorpusID:204402762. Wang, Y ., Dai, P., Jia, X., Zeng, Z., Li, R., and Cao, X. Hi-sigir: Hierachical semantic-guided image-to- image retrieval via scene graph. Proceedings of the 31st ACM International Conference on Multimedia , 2023b. URL https://api.semanticscholar. org/CorpusID:264492649. Wei, J., Narasimhan, H., Am...
work page 2023
-
[12]
URL https://api.semanticscholar. org/CorpusID:255372693. Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? 2019. URL https:// openreview.net/forum?id=ryGs6iA5Km. Yang, J., Ang, Y . Z., Guo, Z., Zhou, K., Zhang, W., and Liu, Z. Panoptic scene graph genera- tion. In European Conference on Computer Vision ,
work page 2019
-
[13]
URL https://api.semanticscholar. org/CorpusID:251018635. Ying, R., Lou, Z., You, J., Wen, C., Canedo, A., and Leskovec, J. Neural subgraph matching. ArXiv, abs/2007.03092, 2020. URL https: //api.semanticscholar.org/CorpusID: 220381221. Yoon, S., Kang, W. Y ., Jeon, S., Lee, S., Han, C., Park, J., and Kim, E.-S. Image-to-image retrieval by learn- ing simil...
arXiv 2007
-
[2017]
URL https://api.semanticscholar. org/CorpusID:3144218. Kipf, T. N. and Welling, M. Variational graph auto- encoders. In Proceedings of NIPS’ Bayesian Deep Learn- ing Workshop, 2016. URL https://arxiv.org/ abs/1611.07308. 9 SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval Krishna, R., Zhu, Y ., Groth, O., Johnson, J., Hata, K., Kr...
arXiv 2016
Show all 16 references
-
[2019]
org/CorpusID:70349949
URL https://api.semanticscholar. org/CorpusID:70349949. Ghosal, S. S. and Li, Y . Are vision transform- ers robust to spurious correlations?, 2024. ISSN 1573-1405. URL https://doi.org/10.1007/ s11263-023-01916-5 . Hasibi, R. and Michoel, T. A graph feature auto- encoder for th...
2024
-
[2021]
org/CorpusID:245445853
URL https://api.semanticscholar. org/CorpusID:245445853. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848. Dimitriou, A., Lymperaiou, M., Filandrianos, G., ...
2009
-
[2022]
org/CorpusID:246411402
URL https://api.semanticscholar. org/CorpusID:246411402. Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. 2023b. URL https://api.semanticscholar. org/CorpusID:256390509. Li,...
2018
-
[2023]
org/CorpusID:267021988
URL https://api.semanticscholar. org/CorpusID:267021988. Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., and Haupt- mann, A. G. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45:1–26,
-
[2024]
Maheshwari, P., Chaudhry, R., and Vinay, V
URL https://openreview.net/forum? id=xkljKdGe4E. Maheshwari, P., Chaudhry, R., and Vinay, V . Scene graph embeddings using relative similarity supervision. Proceedings of the AAAI Conference on Artificial In- telligence, 35(3):2328–2336, May 2021. ISSN 2159-
2021
-
[2025]
ISBN 978-3-031- 72989-8
Springer Nature Switzerland. ISBN 978-3-031- 72989-8. Zhong, X., Yang, Z., Ye, M., Huang, W., Yuan, J., and Lin, C.-W. Auxiliary bi-level graph representation for cross-modal image-text retrieval. 2021 IEEE Interna- tional Conference on Multimedia and Expo (ICME), pp. 1– 6, 20...
2018
-
[5399]
URL http: //dx.doi.org/10.1609/aaai.v35i3.16333
doi: 10.1609/aaai.v35i3.16333. URL http: //dx.doi.org/10.1609/aaai.v35i3.16333. Menon, S., Chandratreya, I. P., and V ondrick, C. Task bias in contrastive vision-language models. In- ternational Journal of Computer Vision , Decem- ber 2023. ISSN 1573-1405. doi: 10.1007/ s11263...
-
[8097]
URL https: //doi.org/10.14778/1687627.1687631
doi: 10.14778/1687627.1687631. URL https: //doi.org/10.14778/1687627.1687631. Zheng, G., Ye, W., and Zhang, A. Benchmarking spurious bias in few-shot image classifiers. pp. 346–364, Cham,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.