{"id":"fcb3e69d-daee-4938-b947-03a0c4c1659d","arxiv_id":"2505.15867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SCENIR, an unsupervised graph autoencoder, ranks images by scene graph similarity and outperforms supervised caption-trained GNNs on a Graph Edit Distance benchmark.","lead":"This paper introduces SCENIR, an unsupervised system that retrieves images by comparing scene graphs, the objects and relationships inside each image, rather than by colors or textures. It claims faster, label-free training and better ranking than supervised graph models, and proposes Graph Edit Distance as a more reliable evaluation standard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GED ground truth is built from the same SBERT class embeddings that form the model's node features, so SCENIR's reported lead may reflect metric alignment rather than semantic retrieval; the metric's validity and stability are unvalidated.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test converges on the same weakest point but sharpens it. The strongest claim depends entirely on the ground-truth ranking. Appendix A reveals that the node feature matrix X is composed of 768-dimensional Sentence-BERT embeddings of object/predicate classes; Appendix B defines GED node substitution costs as cosine distances between exactly those class embeddings. Consequently, the evaluation metric is built from the same representation space that the model consumes and is trained to reconstruct. A method that simply preserves X would rank highly against this metric, so the reported advantage over vision/VL models and even over IRSGS may be a measure of feature-space alignment rather than semantic understanding. This is not a disagreement with consensus; it is an internal circularity risk in the evaluation protocol. The paper also motivates replacing captions because of SBERT variability, but then uses a single unspecified Sentence-Transformer embedding to define the new ground truth, without demonstrating stability across SBERT models; Appendix B does not specify the approximation algorithm or edge costs, and there is a minor 189-versus-133 class-count inconsistency. The unsupervised training and linear-complexity claims are credible and well-supported by the ablations and runtime table; the concern is solely about whether the quantitative evidence establishes semantic superiority. A human-judgment study on a subset, or an exact-GED stability check, would settle this. Because the reader already identified this concern and the conditional verdict is appropriate, I recommend no change.","tokens_in":16512,"tokens_out":6521,"duration_ms":61158,"concrete_test":"On a random sample of 100 PSG test queries, collect blinded human relevance judgments for the top-10 images retrieved by SCENIR-GIN, IRSGS-GIN, and the best vision/VL baseline, and recompute NDCG@k, MAP@k, and MRR using those judgments as ground truth. If SCENIR-GIN does not remain ahead of IRSGS-GIN, the central semantic-superiority claim is an artifact of the SBERT-based GED metric rather than a genuine semantic advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SCENIR achieves superior semantic image-to-image retrieval. Every quantitative result in Tables 1 through 4 is measured against a ground-truth ranking derived from approximate Graph Edit Distance (Appendix B), but that GED is not an independent semantic measure. Node substitution costs are cosine distances between Sentence-BERT embeddings of object/predicate class names, and the same 768-dimensional Sentence-BERT embeddings are used as the node feature matrix X supplied to every GNN (Appendix A). SCENIR is trained to reconstruct X and A, so a model that preserves its input features will by construction score well against a metric computed from those same features. This does not establish that SCENIR captures human-perceived semantics; it establishes alignment with one particular SBERT-derived similarity. The paper explicitly criticizes SBERT caption variability (Fig. 2) yet replaces captions with GED costs drawn from an unspecified Sentence-Transformer model, without showing that class-name embeddings are stable across SBERT variants or that cosine-distance GED approximates exact GED or human relevance. Appendix B defines node substitution and insertion/deletion costs but does not specify the GED approximation algorithm or edge edit costs, despite exact GED being NP-hard and 1,000 test graphs requiring roughly 500k pairwise scores. The comparison with supervised IRSGS is also potentially biased: IRSGS is trained on caption-based SBERT similarity labels, while SCENIR's inputs and the evaluation metric share the same SBERT class-embedding space. There is also a numerical inconsistency (189 classes in Appendix A versus 133 in Appendix B). These issues make the headline margin (31.39 versus 29.64 NDCG@1) insufficiently supported as evidence of superior semantic retrieval.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SCENIR, an unsupervised graph autoencoder for image-to-image retrieval via scene graphs. SCENIR encodes scene graphs with split GNN encoder branches, reconstructs node features and adjacency with MLP decoders, and applies adversarial regularization; retrieval is performed by cosine similarity on sum-pooled latent embeddings. The authors evaluate on PSG against vision, vision-language, supervised GNN (IRSGS), and GAE baselines, using approximate Graph Edit Distance over 1K test graphs as ground truth. They report superior retrieval metrics for SCENIR-GIN, linear preprocessing/training/inference complexity, and additional experiments on Flickr30K and counterfactual retrieval. The GitHub repository containing the source code is provided.","tokens_in":16786,"tokens_out":6603,"duration_ms":56771,"significance":"If the results hold, the paper makes a useful contribution: it demonstrates that unsupervised graph autoencoders can be competitive with or superior to supervised scene-graph retrieval, and it advocates a deterministic GED-based evaluation protocol. The efficiency gain (8 minutes versus 50 minutes for IRSGS, and versus 3 hours for GC) and the inductive counterfactual application are concrete strengths. However, the central empirical claim depends on the validity of the GED ground truth and on the statistical robustness of the comparisons; both require additional support before the significance can be fully assessed.","major_comments":[{"comment":"The evaluation metric is built from the same Sentence-BERT class embeddings that serve as the model's input node features. Appendix A states that X uses 768-dimensional Sentence-Transformer embeddings of the 189 object and predicate classes, and Appendix B defines the GED node substitution cost as the cosine distance between exactly those embeddings. Because SCENIR's loss includes feature reconstruction (Eq. 3, L_featrecon), a model that preserves its input features will by construction rank well under this GED. This makes every reported metric in Tables 1-4 partially endogenous to the model's input representation, so the results may reflect alignment with an SBERT-derived similarity rather than semantic relevance. To support the semantic-retrieval claim, the authors should validate the GED ranking against human judgments, test stability across multiple Sentence-Transformer models, and compare against exact GED or a well-established approximation.","section":"3.1, Appendix A and B"},{"comment":"The strongest quantitative claims are not supported by the reported numbers. In Table 1, SCENIR-GIN has MRR 59.01, while IRSGS-GIN with RoBERTa supervision has MRR 59.16, so SCENIR is not superior on MRR; NDCG@10 is essentially tied (26.28 vs. 26.00). The text in Section 4.1 claims SCENIR \"surpasses even the supervised IRSGS across most metrics,\" but the abstract and conclusion claim \"superior performance across metrics,\" which is contradicted by Table 1. Since the paper reports no error bars, multiple seeds, or significance tests, even the clearer gaps (e.g., NDCG@1 31.39 vs. 29.64) cannot be distinguished from run-to-run variation. The authors should report means and standard deviations over several runs, perform significance tests, and adjust the strength of their claims accordingly.","section":"4.1, Table 1"},{"comment":"The description of the GED approximation is incomplete and does not support the paper's claim that GED is a \"deterministic and robust ground truth.\" Exact GED is NP-hard (as cited in Section 2), yet the authors compute pairwise GED scores for 1,000 test graphs, yielding roughly half a million comparisons; they do not specify the approximation algorithm, the edge edit costs, or the accuracy of the approximation relative to exact GED. Without this information, the reader cannot assess whether the ground-truth rankings are reliable or how sensitive they are to the approximation's parameters. The authors should provide the algorithm, its parameters, and a validation study on a sample of graphs.","section":"Appendix B"},{"comment":"No validation split is described. The loss trade-off weights (lambda1=3, lambda2=1/6, lambda3=1/3) and the 3-layer encoder are selected through performance sweeps reported in Appendix D (Figures 8 and 9). If these sweeps were performed on the same 1K test graphs used to produce Tables 1-4, the reported results are optimistically biased and the comparisons are not statistically valid. The authors must clarify whether model selection used a separate validation set and, if so, provide its size and construction; otherwise, the hyperparameter tuning is effectively performed on the test set.","section":"4 and Appendix D"},{"comment":"The comparison with IRSGS is asymmetric because IRSGS is trained on caption-based SBERT similarity labels while the evaluation uses GED. This is a meaningful design choice, since the paper argues that caption supervision is inconsistent, but it means Table 1 does not establish that SCENIR learns GED better; it establishes that caption-trained models transfer worse to a GED-based metric. To make the comparison more conclusive, the authors should either include an IRSGS variant trained with GED-based labels, report the caption-based evaluation as well, or explicitly frame the result as a transfer comparison between training signals rather than as a direct superiority claim.","section":"4.1, Baselines"}],"minor_comments":[{"comment":"DeiT is a vision transformer, not a vision-language model; the caption \"VL (DeiT)\" and the text \"In Figure 6, DeiT, a VL model\" are incorrect and should say \"Vision.\"","section":"4.2, Figure 6 caption"},{"comment":"There is a typo: \"SCENIR achievessignificant\" should be \"SCENIR achieves significant.\" Also, \"SotA\" is used as an adjective inconsistently throughout the paper; consider spelling out \"state-of-the-art\" on first use.","section":"4.4"},{"comment":"The first sentence references \"the qualitative results of Figures 5 and 6 of the main paper,\" but the qualitative results appear in Figures 6 and 7; the cross-reference should be corrected.","section":"Appendix E"},{"comment":"The main-text Figure 5 is labeled as NDCG@5 versus number of GNN layers, while Appendix D Figure 8 reports NDCG@3, MAP@3, and MRR for the same layer sweep; this inconsistency could confuse readers and should be reconciled.","section":"Main text Figure 5 and Appendix D Figure 8"},{"comment":"The Sentence-Transformer model used to construct the 768-dimensional class embeddings is not specified (no model name or version). Since the paper criticizes SBERT variability, the evaluation's dependence on one particular embedding model should be stated explicitly and justified.","section":"Appendix A"},{"comment":"The citation \"Puaduraru et al., 2024\" appears to contain a misspelling (likely \"Pudararu\"); please verify the reference.","section":"Introduction references"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the code release is a positive feature. The main risk is that the empirical claims rest on a GED ground truth that is partially derived from the same SBERT embeddings that the model consumes, and on single-run comparisons with no significance testing. If the authors can validate the GED metric against human judgments or an independent semantic measure, and provide a sound statistical analysis, the contribution could be solid. Otherwise the headline result may reduce to alignment with a specific SBERT-based similarity measure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SCENIR is a sensible unsupervised graph-autoencoder pipeline for scene graph retrieval, and the authors are right that caption-based supervision is fragile. But the evaluation metric is built from the same Sentence-BERT class embeddings that serve as the model's node features, so the headline NDCG lead over IRSGS is, at this point, evidence of alignment with a specific embedding space rather than of semantic retrieval.\n\nWhat's actually new: this is the first unsupervised scene graph retrieval framework I know of. The architecture combines existing pieces—split VGAE encoder, MLP decoders, adversarial regularization—but the combination is nontrivial and the ablations are informative. The linear complexity and 8-minute training time are real practical advantages. I also credit them for explicitly advocating GED as an evaluation protocol, even if their version is approximate.\n\nSoft spots: the stress-test concern is legitimate. Appendix A says node features X are 768-d SBERT embeddings of 189 object/predicate classes; Appendix B defines GED node substitution cost as cosine distance between node embeddings, with insertion/deletion cost relative to the mean of 133 class embeddings. So the ground truth and the model input share the same embedding space. SCENIR is trained to reconstruct X, so it will score well against a metric derived from X. That doesn't make the training circular—they never fit GED labels—but it makes the metric partially endogenous and unvalidated as a proxy for human semantics. They criticize SBERT caption instability yet never check whether class-name embeddings are stable across SBERT variants, and they don't specify the GED approximation algorithm or edge edit costs. Also: no error bars, no validation split (Appendix D tunes lambda and layer count on what looks like the test set), and a numerical inconsistency (189 classes vs 133). These are all addressable.\n\nWho it's for: people working on scene graph retrieval, graph autoencoders, and counterfactual evaluation. I'd send it to review—the method is novel enough and the flaws are fixable—but the next version needs multi-seed results, a separate validation split, and a sanity check of the GED approximation against exact GED or human judgments.\n\nRecommendation: engage with it, but don't take the performance claims at face value until the evaluation is tightened.","headline":"SCENIR is a sensible unsupervised GAE for scene graph retrieval, but its GED evaluation metric shares the same SBERT embedding space as the model's input features, so the headline performance lead is not yet convincing evidence of semantic retrieval.","tokens_in":17421,"tokens_out":2835,"would_cite":false,"duration_ms":25959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised graph autoencoder can rank images by semantic content, beating caption-supervised graph models and vision models.","keywords":["scene graph retrieval","graph autoencoder","unsupervised learning","graph edit distance","image-to-image retrieval","semantic similarity","graph neural networks","counterfactual retrieval"],"falsifier":"Take a random sample of PSG image pairs, have humans rate semantic similarity, and compare those ratings with the approximate GED rankings used here; low rank correlation would show the evaluation oracle, and with it the central claim, does not capture human semantics. A cheaper partial falsifier is to compute exact GED on small graphs and measure how often the approximation used in the paper changes the top-k ranking.","tokens_in":16292,"feed_emoji":"🖼️","tokens_out":5549,"duration_ms":46195,"temperature":0.7,"pith_summary":"This paper argues that image-to-image retrieval should rank images by their semantic content—objects, attributes, and the relations between them—rather than by low-level visual features such as color, and that scene graphs are the right carrier for that semantics. It introduces SCENIR, a graph autoencoder trained with no labels at all, which embeds scene graphs into a latent space and retrieves by cosine similarity. Against ground-truth rankings computed by Graph Edit Distance, the paper reports that SCENIR outperforms vision-only models, vision-language models, and the previously leading supervised scene-graph retriever, while running the full pipeline in about eight minutes instead of about fifty. The paper further proposes Graph Edit Distance itself as the evaluation standard, replacing caption-based similarity, which it shows to be unstable across Sentence-BERT variants.","feed_headline":"Unsupervised graph autoencoder beats caption-trained retrieval","feed_subtitle":"A label-free scene graph model ranks images by meaning, not color, and runs in about 8 minutes end to end.","key_machinery":"The load-bearing object is the SCENIR graph autoencoder: a variational graph encoder with two independent three-layer GNN branches (one for mean, one for variance), dual MLP decoders that reconstruct the adjacency matrix and the node feature matrix, and an adversarial discriminator that regularizes the latent space toward a Gaussian prior. At inference the mean branch's node embeddings are sum-pooled to a graph vector, and retrieval is cosine ranking. Graph Edit Distance, with node substitution costs defined as cosine distance over 768-dimensional class embeddings, serves as the deterministic evaluation oracle that all reported metrics are computed against.","core_discovery":"The central claim is that a fully unsupervised graph autoencoder can learn scene graph embeddings that support more semantically faithful image retrieval than supervised alternatives, provided the architecture is adjusted to scene graphs. SCENIR splits the variational encoder into independent mean and variance GNN branches, decodes with two parallel 2-layer MLPs that reconstruct both adjacency and node features, and adds adversarial regularization; at inference it sum-pools the mean latent node embeddings and ranks candidates by cosine similarity. On the PSG dataset, the best configuration (with a GIN backbone) reaches NDCG@1 of 31.39 versus 29.64 for the best supervised IRSGS variant, and the unsupervised model remains the top ranker on in-the-wild Flickr30K images whose scene graphs are generated automatically. The paper also asserts that Graph Edit Distance, approximated with embedding-based node substitution costs, gives a deterministic ground truth for evaluating scene graph retrieval, replacing the caption-based supervision that shifts with the choice of sentence encoder.","pith_inferences":["Extension: if GED with cosine node costs is replaced by human-annotated semantic similarity, the relative ranking of methods is an open empirical question; the paper's own strongest claim is specifically about matching its GED-derived rankings.","The success of a simple 3-layer GIN encoder plus MLP decoders suggests that over-smoothing, not depth, is the binding constraint for scene graphs, so deeper architectures may pay off if skip connections or normalizers are added.","Because retrieval happens in embedding space, the same SCENIR encoder could be plugged into text-to-image or cross-modal pipelines by aligning scene graph embeddings with text embeddings, a direction the paper does not pursue."],"forward_implications":["Image retrieval can be built without caption or similarity labels, removing the dependency on text encoders whose disagreements propagate into trained retrievers.","Since preprocessing, training, and inference are all linear in dataset size, the approach scales to graph corpora where quadratic label construction would be prohibitive.","Using one deterministic graph-distance ground truth makes retrieval results reproducible and comparable across research groups, unlike caption-based rankings.","The same embeddings support downstream uses such as counterfactual image explanations, where SCENIR outperforms a supervised counterfactual framework in an inductive setting."],"supporting_citations":[{"why":"Supplies the supervised IRSGS scene graph retrieval baseline that SCENIR must beat and the caption-based similarity setup the paper argues against.","marker":"(Yoon et al., 2021)"},{"why":"Provides the approximate Graph Edit Distance evaluation convention and the counterfactual scene graph framework that SCENIR extends and compares against.","marker":"(Dimitriou et al., 2024)"},{"why":"Defines the variational graph autoencoder architecture, including the split-encoder design and variational loss that SCENIR builds on.","marker":"(Kipf & Welling, 2016)"},{"why":"Supplies the adversarial regularization component and training algorithm that SCENIR adopts for latent space shaping.","marker":"(Pan et al., 2018)"},{"why":"Motivates the GIN backbone and sum-pooling strategy, and gives the expressiveness argument for why GIN variants outperform GCN variants.","marker":"(Xu et al., 2019)"},{"why":"Defines Graph Edit Distance, the deterministic similarity measure the paper proposes as evaluation ground truth.","marker":"(Sanfeliu & Fu, 1983)"},{"why":"Provides the Sentence-BERT caption embeddings whose variability across models motivates replacing caption-based supervision with GED.","marker":"(Reimers & Gurevych, 2019)"},{"why":"Supplies the PSG dataset, including its scene graphs, captions, and the PSGTR scene graph generator used in the in-the-wild experiment.","marker":"(Yang et al., 2022)"}],"fun_headline_variants":["SCENIR: label-free scene graphs win image retrieval","Unsupervised scene graph retrieval tops supervised methods","No captions needed: SCENIR beats supervised retrieval","Graph autoencoder redefines image search without labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking that all results depend on treats approximate Graph Edit Distance, with node costs derived from Sentence-BERT class embeddings, as the true measure of semantic similarity between images; if that measure does not track what people mean by semantic similarity, the reported superiority is superiority at this metric.","fun_headline_variants_meta":{"raw":{"variants":["SCENIR: label-free scene graphs win image retrieval","Unsupervised scene graph retrieval tops supervised methods","No captions needed: SCENIR beats supervised retrieval","Graph autoencoder redefines image search without labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3430,"prompt_tokens":959,"completion_tokens":2471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2407}},"tokens_in":575,"tokens_out":2471,"duration_ms":15415,"temperature":1.0,"reasoning_tokens":2407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:17:55.559940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of PSG image pairs, have humans rate semantic similarity, and compare those ratings with the approximate GED rankings used here; low rank correlation would show the evaluation oracle, and with it the central claim, does not capture human semantics. A cheaper partial falsifier is to compute exact GED on small graphs and measure how often the approximation used in the paper changes the top-k ranking.","supporting_citations":[],"review_version":1}