{"id":"4fe87ab7-aa40-4e5e-a1fe-2ee06f3ec548","arxiv_id":"2507.05798","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SPADE uses DDIM inversion-derived cross-attention maps to calibrate a LoRA-tuned diffusion UNet and a relation graph transformer to improve open-vocabulary panoptic scene graph generation.","lead":"This paper introduces SPADE, a panoptic scene graph generation model that adapts a diffusion model's inversion cross-attention maps to improve spatial relation reasoning in open-vocabulary settings. It reports gains over prior open-set PSG and SGG models on PSG and Visual Genome benchmarks, especially for spatial predicates and distant object pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) calibration loss compares cross-attention maps of different token dimensions (text tokens vs. CLIP image features), leaving the central mechanism underspecified and unverifiable.","rationale":"We judge the dimensional mismatch of Eq. (6) to be the most load-bearing concern because it attacks the paper's proposed mechanism, not just its experimental reporting. If A_i and A'_i have incompatible token dimensions, then the calibration loss in its literal form cannot be evaluated; a working implementation must silently insert a projection (e.g., truncating text tokens, repeating the CLIP token, or pooling), and each choice changes what spatial prior is distilled. Since the paper's novelty is precisely this inversion-guided calibration, and the ablations in Table 5 attribute large gains (e.g., 15.3 to 26.7 R@50 on OvR) to it, an unstated alignment step leaves the core claim unsupported and not independently reproducible.\n\nThe reader's verdict (CONDITIONAL) is appropriate. We do not recommend REJECT because the empirical numbers could still be correct even if the calibration mechanism is more ad hoc than described; the concern is a correctness/reproducibility risk rather than a demonstrated contradiction. We also note the other weaknesses listed by the reader (missing inversion sampler, LoRA rank, prompt format, lambda, single-run results), but those are secondary—they could be fixed in a revision. The Eq. (6) shape mismatch is the one that makes the method as written non-executable.\n\nAgreement with reader: we agree with the reader's weakest_assumption that the alignment of the two cross-attention maps is unexamined; our formulation strengthens it from 'semantic compatibility' to 'dimensional compatibility,' which is the first thing that must be checked.\n\nThe concrete test would settle the concern by checking shapes and then testing whether the ablation gain survives a specified alignment. If shapes already match (e.g., CLIP patch count equals text token count after a prompt template with exactly that many placeholders, which is implausible), the concern is resolved. If shapes match only after a projection, the paper should describe it and the test would reveal whether the reported gains depend on that choice.","tokens_in":16411,"tokens_out":5869,"duration_ms":58385,"concrete_test":"Run a single forward pass on one PSG training image with the released configuration and print torch.size(A_i) and torch.size(A'_i) at the calibration loss. If the token dimensions differ (L_cond != L_text), Eq. (6) is undefined as written; if the authors used a projection, the paper must specify it. Then default the projection to 'teacher prompt tokens' and compute the per-token L1 loss; if the Table 5 gap between 'w/o calibration' (15.3 R@50) and 'Ours' (26.7) shrinks by more than 5 points, the claimed benefit depends on the unstated alignment rather than on spatial inversion priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SPADE's gains come from aligning student cross-attention maps with teacher inversion maps—rests on Eq. (6), but the two maps cannot have the same shape under the paper's own definitions. The teacher map A'_i is produced by conditioning the frozen UNet on the text prompt p = '[subject] is [predicate] [object]' via the CLIP text encoder (§3.1), so A'_i has a token dimension equal to the prompt length (typically 77). The student map A_i is produced by conditioning the same UNet on MLP(CLIP_image(x_i)) (Eq. (4)). CLIP_image outputs a class token (sequence length 1) or a fixed patch grid (e.g., 196 or 256 tokens), not 77 tokens. The paper gives no pooling, padding, or projection that maps one sequence length to the other. Without such an alignment, ||A_i - A'_i||_1 in Eq. (6) is undefined, and any implementation must introduce an unspecified transformation that changes the spatial prior being distilled. This makes the central mechanism irreproducible as described and weakens the attribution of the Table 1 and Table 5 gains to inversion-guided calibration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SPADE is a two-stage framework for open-vocabulary panoptic scene graph generation. Stage one calibrates a pre-trained text-to-image diffusion UNet into a PSG-specific denoising network: DDIM inversion of real images with relation prompts yields teacher cross-attention maps, and a calibration loss aligns student cross-attention maps to these under LoRA adaptation of the UNet. Stage two constructs a spatial-semantic graph from instance masks and applies a relation graph transformer with long-range neighbor/non-neighbor attention and graph-convolutional local correlation, followed by relation-query construction and prompt-based open-vocabulary classification that fuses diffusion and pooled CLIP features. On PSG and VG, the paper reports state-of-the-art closed-set and open-set results, with the largest margins on spatial predicate prediction and distant relation pairs.","tokens_in":16661,"tokens_out":6204,"duration_ms":67515,"significance":"The paper addresses a genuine weakness of VLM-based PSG, namely spatial reasoning at distance, with a novel and plausible mechanism. The empirical study is broad: two datasets, closed-set/OvR/OvD+R settings, component ablations, and a deliberate seen-only calibration split for open-set experiments. If the inversion-guided calibration is made well-defined, the method is a useful contribution. As written, however, the central loss in Eq. (6) is underspecified to the point of being undefined, so the significance currently rests on an unclear mechanism.","major_comments":[{"comment":"The calibration loss is central to the method, but the two cross-attention maps it aligns have different conditioning modalities and token dimensions. The student map A_i is computed with MLP∘CLIP_image(x_i) as the UNet conditioning signal (Eq. (4)), while the teacher map A'_i is computed from the text prompt p='[subject] is [predicate] [object]' (§3.1). CLIP image features and text-token embeddings do not share a sequence length, and no pooling, padding, or projection is specified. As written, ||A_i - A'_i||_1 is undefined. Please specify the exact alignment used and explain why the aligned maps are semantically comparable enough to serve as spatial priors.","section":"§3.1, Eq. (6)"},{"comment":"The attention equations for long-range correlation learning are not well-formed. The expression contains 'p |P(r)+|' in the numerator without a division or a defined symbol p, and the scaling convention is missing. The notation φ[P(r)+] for averaging neighbor features is also not formally defined. Since RGT is the core context-reasoning module, these equations need to be rewritten with clear tensor shapes and scaling.","section":"§3.2, Eqs. (8)-(9)"},{"comment":"The prompt-based classification formulas are dimensionally unclear: dividing the feature-transpose product by ϵ_t(T_o), a text embedding, is not a valid operation as written. Presumably the intended form is a scaled cosine similarity or inner product between H and text embeddings with a temperature, but the notation prevents the reader from knowing. Please rewrite these equations and define the shapes of all tensors.","section":"§3.3, Eqs. (16)-(17)"},{"comment":"The 'w/o inversion' ablation, which replaces deterministic inversion with random Gaussian sampling, is central to attributing the gain to inversion guidance, but the description does not state whether the teacher cross-attention maps are then computed from random-noise sampling or whether the student UNet is trained with the same LoRA recipe. Please clarify the exact protocol; otherwise the 21.0→26.7 gain cannot be assigned to the inversion prior.","section":"§4.3, Table 5"}],"minor_comments":[{"comment":"The default diffusion timestep is t=0, but the inversion and calibration section implies features from the 'final timestep t'; please state the inversion step count and which timesteps produce the teacher maps.","section":"§4.1"},{"comment":"The denominator |ˆQr||ˆQr^T| is not defined as a matrix norm; specify the norm and the exact construction of Ψ'.","section":"Eq. (14)"},{"comment":"It is unclear whether Dis is a distance (smaller=closer) or similarity (larger=closer) given Select> η; please clarify.","section":"Eq. (13)"},{"comment":"The DR/NDR split is based on one-third image width; state whether this uses normalized coordinates and how border cases are treated.","section":"§4.1"},{"comment":"No variance or significance tests are reported for the main tables; since the margins over OpenPSG are 2–5 points, multiple seeds or standard deviations would increase confidence.","section":"General"},{"comment":"Typographical issues: 'reversion process' in the contribution list, 'Lmaks' for L_mask, the project page link 'at here', and duplicate GLIP references [28]/[29] should be corrected.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is whether the authors can supply a well-defined alignment for Eq. (6) without changing the method. If they can, the empirical contribution is solid; if not, the headline gains are not attributable to inversion-guided calibration. I recommend asking for the exact implementation and a re-run or clarification of Table 5 with the alignment explicitly instantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SPADE is a genuinely new combination for open-vocabulary PSG: it distills cross-attention maps from DDIM inversion into a LoRA-tuned UNet and then runs a relation graph transformer over the resulting features. That is a fresh angle on a real weakness — CLIP-based models are known to be poor at spatial and distant relations, and the paper documents that gap clearly in Table 4. The reported gains over OpenPSG and OvSGTR on PSG and VG are consistent across closed- and open-set settings, and the ablations (Tables 5–7) show each component contributes. I find the central empirical claim plausible.\n\nThe soft spots are real but not fatal. The biggest issue is Eq. (6). The calibration loss writes ||A_i - A'_i||_1, but the teacher map comes from a text-conditioned prompt while the student map is conditioned on CLIP image features. The token dimensions do not obviously match, and the paper gives no pooling, padding, or projection to align them. The stress-test note is right: as written, the loss is underspecified and the central mechanism is not reproducible. This is exactly the kind of detail a referee needs to pin down. It could well be an oversight with a simple fix, but it needs to be stated.\n\nSecond, all results are single-run with no variance or significance tests, yet the text calls gains 'significant.' That is a style issue but should be fixed. Third, several hyperparameters (alpha, eta, lambda, LoRA rank) are fixed without sensitivity analysis. The paper should at least provide a small sweep or justification. Eqs. (8), (16), and (17) also have notation and dimension ambiguities, but those are minor and easily clarified.\n\nWho is this for? Researchers working on open-vocabulary scene graph generation and anyone trying to inject spatial priors into diffusion-based perception. If the calibration-loss detail is clarified and code is released, this would be a solid contribution. Even without code, the paper deserves a serious referee because the idea is novel and the experiments are carefully done on standard benchmarks.\n\nMy recommendation: send to peer review with a request for major revision — clarify Eq. (6) alignment, add implementation details (inversion sampler, LoRA rank, prompt format), and either include error bars or tone down the 'significant' language.","headline":"A useful new combination for open-vocabulary PSG with a clear spatial-prior idea, but the central calibration loss is underspecified and needs a major revision.","tokens_in":17222,"tokens_out":4405,"would_cite":false,"duration_ms":45704,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPADE injects spatial awareness into diffusion-based scene graph generation and outperforms prior open-set models.","keywords":["panoptic scene graph generation","open-vocabulary scene graph generation","spatial relation reasoning","diffusion model inversion","DDIM inversion","cross-attention distillation","LoRA fine-tuning","relation graph transformer"],"falsifier":"Run the identical SPADE pipeline but condition the teacher inversion on a non-spatial prompt such as 'a photo' instead of '[subject] is [predicate] [object]'. If open-set recall on PSG stays high, the spatial prompt is not carrying the result; if recall drops to baseline, the inversion geometry alone is insufficient and the calibration signal is doing the work.","tokens_in":16139,"feed_emoji":"🕸️","tokens_out":5744,"duration_ms":55899,"temperature":0.7,"pith_summary":"The paper argues that open-vocabulary panoptic scene graph generation models built on vision-language models fail at spatial relation reasoning, especially when the subject and object are far apart in the image. SPADE is a two-stage remedy: first calibrate a pre-trained diffusion model into a PSG-specific denoising network by aligning its inversion-derived cross-attention maps with those of a LoRA-tuned student UNet; then run a relation graph transformer that mixes long-range and local context to build relation queries. On the PSG and Visual Genome benchmarks, the paper reports state-of-the-art closed-set and open-set results, with the largest gains on spatial predicates and distant relation pairs.","feed_headline":"Scene graph model gains spatial sense from diffusion inversion","feed_subtitle":"On PSG and VG, open-set recall climbs to 26.7% and distant-pair spatial relations nearly catch up with nearby ones.","key_machinery":"The load-bearing object is the calibration loss of Eq. (6), an L1 alignment between two cross-attention maps: $A_i$ produced by the student UNet conditioned on CLIP image features through an MLP adapter, and $A'_i$ produced by a frozen teacher diffusion model conditioned on the prompt '[subject] is [predicate] [object]' during deterministic DDIM inversion, the process that maps an image to noise while preserving its spatial layout. This loss is the only conduit through which spatial priors enter the UNet; it is paired with LoRA updates restricted to cross-attention layers to avoid destroying the pretrained model. The second stage, a relation graph transformer with long-range neighbor/non-neighbor self-attention and a GCN local block, builds pairwise relation queries.","core_discovery":"SPADE's central claim is that the spatial structure a DDIM inversion preserves can be transferred into a PSG model without sacrificing the open-world knowledge of the frozen diffusion and CLIP backbones. Using cross-attention maps from the teacher's inversion as spatial priors, the method calibrates a UNet via rank-limited updates, then fuses diffusion features with pooled CLIP features for open-vocabulary classification. A relation graph transformer captures both connected and non-connected object context. On PSG closed-set, R/mR@50 reaches 45.1/41.2 versus 42.8/38.9 for OpenPSG, and open-set OvR R/mR@50 reaches 26.7/23.3 versus 21.2/19.8; on VG, open-set OvR R/mR@50 reaches 24.1/11.2 versus 20.4/9.4.","pith_inferences":["The calibration loss treats cross-attention maps as a spatial prior, but the paper does not establish what those maps mean semantically; a natural test is whether the alignment transfers spatial layout rather than just matching global statistics.","The method's dependence on the prompt template '[subject] is [predicate] [object]' may inherit the teacher's own spatial reasoning limits, so relations not expressible in that form are a likely failure mode.","The graph transformer selects object pairs by cosine distance above a fixed threshold; making that threshold adaptive per image is a testable extension the paper leaves open.","If the gains come mostly from the calibration alignment, a lighter alternative that distills inversion maps directly into a non-diffusion encoder might capture similar spatial priors at lower inference cost."],"forward_implications":["Spatial predicate prediction in open-vocabulary PSG can be improved without retraining the VLM or diffusion backbone, by supervised alignment to inversion cross-attention maps.","Distant relation pairs, which current VLM models handle poorly, become a targetable weakness: the calibration plus graph transformer nearly closes the gap between distant and non-distant mR@50 (38.7 vs 42.8 on PSG).","The same two-stage recipe could transfer to other structured prediction tasks that need spatial awareness, such as human-object interaction detection, since no task-specific spatial head is assumed beyond the graph transformer.","The reported gains are consistent across two datasets (PSG and VG) and across closed-set, OvR, and OvD+R settings, suggesting the mechanism generalizes beyond a single benchmark."],"supporting_citations":[{"why":"Provides the DDIM deterministic inversion and sampling process whose cross-attention maps are used as spatial priors.","marker":"[45]"},{"why":"Supplies the exact-inversion sampler used as the teacher model for calibration.","marker":"[50]"},{"why":"The pre-trained latent diffusion model and UNet backbone that SPADE calibrates.","marker":"[39]"},{"why":"Supplies the LoRA low-rank fine-tuning method that preserves pretrained knowledge during calibration.","marker":"[19]"},{"why":"Open-vocabulary scene graph generation baseline whose prompt-based classifier protocol SPADE extends and compares against.","marker":"[4]"},{"why":"The main open-set PSG baseline that SPADE outperforms in Tables 1, 3, and 4.","marker":"[69]"},{"why":"Prior use of text-to-image diffusion features for open-vocabulary segmentation; source of the CLIP-image conditioning and geometric mean fusion.","marker":"[56]"},{"why":"VLM-based open-set PSG baseline used in the spatial relation analysis and main comparisons.","marker":"[31]"},{"why":"Defines the PSG dataset, task, and mask/relation losses used for training and evaluation.","marker":"[57]"},{"why":"Provides the Visual Genome dataset used for the second benchmark and calibration.","marker":"[24]"}],"fun_headline_variants":["Diffusion inversion improves open-vocabulary scene graph relations","Spatial-aware denoising network sharpens scene graph relations","Open-vocabulary scene graph generation gets spatial sense from diffusion","SPADE: diffusion inversion teaches scene graph models spatial relations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration loss assumes that the cross-attention maps produced by the student UNet and the teacher diffusion model are comparable L1-alignable quantities with matching spatial and semantic structure; the paper does not establish that the two maps are compatible.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion inversion improves open-vocabulary scene graph relations","Spatial-aware denoising network sharpens scene graph relations","Open-vocabulary scene graph generation gets spatial sense from diffusion","SPADE: diffusion inversion teaches scene graph models spatial relations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00105,"raw_usage":{"total_tokens":4433,"prompt_tokens":992,"completion_tokens":3441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":3371}},"tokens_in":608,"tokens_out":3441,"duration_ms":27258,"temperature":1.0,"reasoning_tokens":3371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:18:17.147056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical SPADE pipeline but condition the teacher inversion on a non-spatial prompt such as 'a photo' instead of '[subject] is [predicate] [object]'. If open-set recall on PSG stays high, the spatial prompt is not carrying the result; if recall drops to baseline, the inversion geometry alone is insufficient and the calibration signal is doing the work.","supporting_citations":[{"cited_title":"Denois- ing diffusion implicit models","cited_arxiv_id":null,"evidence_quote":"Provides the DDIM deterministic inversion and sampling process whose cross-attention maps are used as spatial priors."},{"cited_title":"Belm: Bidirec- tional explicit linear multi-step sampler for exact inversion in diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the exact-inversion sampler used as the teacher model for calibration."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"The pre-trained latent diffusion model and UNet backbone that SPADE calibrates."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA low-rank fine-tuning method that preserves pretrained knowledge during calibration."},{"cited_title":"Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention","cited_arxiv_id":null,"evidence_quote":"Open-vocabulary scene graph generation baseline whose prompt-based classifier protocol SPADE extends and compares against."},{"cited_title":"Openpsg: Open-set panoptic scene graph generation via large multimodal models","cited_arxiv_id":null,"evidence_quote":"The main open-set PSG baseline that SPADE outperforms in Tables 1, 3, and 4."},{"cited_title":"Open-vocabulary panop- tic segmentation with text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Prior use of text-to-image diffusion features for open-vocabulary segmentation; source of the CLIP-image conditioning and geometric mean fusion."},{"cited_title":"From pixels to graphs: Open-vocabulary scene graph generation with vision-language models","cited_arxiv_id":null,"evidence_quote":"VLM-based open-set PSG baseline used in the spatial relation analysis and main comparisons."},{"cited_title":"Panoptic scene graph gen- eration","cited_arxiv_id":null,"evidence_quote":"Defines the PSG dataset, task, and mask/relation losses used for training and evaluation."},{"cited_title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","cited_arxiv_id":null,"evidence_quote":"Provides the Visual Genome dataset used for the second benchmark and calibration."}],"review_version":1}