Pith. sign in

REVIEW 4 major objections 8 minor 52 references

Two reasoning modes, one decoder: Dual-SGG unifies scene graph generation

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A dual-query scene graph generation method unifies detector-based and query-based reasoning in a single decoder, achieving state-of-the-art results on Visual Genome, Open Images v6, and GQA-200.

T0 review reviewed 2026-07-08 challenge →

load-bearing objection Dual-SGG combines detector-based and query-based SGG reasoning in one decoder via dual queries; achieves SOTA on VG, OIv6, and GQA-200 with a useful diagnostic framework for understanding complementarity between the two paradigms. the 4 major comments →

arxiv 2607.06176 v1 pith:2CVY6D4W submitted 2026-07-07 cs.CV

Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability

Runfeng Qu , Pia K Bideau , Ole Hall , Julie Ouerfelli-Ethier , Klaus Obermayer , Olaf Hellwich This is my paper
classification cs.CV
keywords scene graph generationdetector-conditioned reachabilitydual-query designquery-based reasoningdetector-based reasoningtriplet predictionvisual relationship detectiontransformer decoder
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Scene graph generation (SGG) — the task of extracting (subject, predicate, object) triplets from images — has been split between two paradigms. Detector-based methods first find objects, then reason over pairs, but cannot predict relationships involving objects the detector missed. Query-based methods learn global triplet queries that bypass the detector entirely, recovering some of those missed relationships, but at the cost of accuracy on the ones the detector does find. This paper introduces the concept of detector-conditioned reachability: a formal way to split ground-truth triplets into those whose subject and object are both detected by an external object detector (Det-T) and those where at least one is missed (UDet-T). Using this split, the authors show empirically that detector-based models dominate on Det-T but nearly collapse on UDet-T, while query-based models recover UDet-T triplets but lose ground on Det-T. This complementarity is the paper's central empirical finding. To exploit it, they propose Dual-SGG, which places two groups of triplet queries in a single transformer decoder: top-down triplet queries (TD-Qs) initialized from detected entity pairs to preserve detector-conditioned reasoning, and bottom-up triplet queries (BU-Qs) initialized from image-center anchors to explore globally. Because both query types are updated jointly in one decoder, their outputs share a common score scale and require no post-hoc calibration. NMS removes duplicates. The method achieves state-of-the-art results on Visual Genome, Open Images v6, and GQA-200 at 14 FPS.

Core claim

The paper's central discovery is that the predictive behaviors of detector-based and query-based SGG methods are complementary when analyzed through the lens of detector-conditioned reachability. Detector-based models achieve high recall on triplets whose entities are detected (Det-T) but near-zero recall on triplets with at least one undetected entity (UDet-T). Query-based models partially recover UDet-T triplets but sacrifice Det-T performance. By integrating both reasoning mechanisms in a single decoder via a dual-query design — TD-Qs conditioned on detected entity pairs and BU-Qs exploring globally from center-biased anchors — Dual-SGG preserves the strengths of both paradigms without a

What carries the argument

The detector-conditioned reachability split (Det-T vs UDet-T) using an external object detector as a discriminator; the dual-query design with top-down triplet queries (TD-Qs) initialized from selected entity pairs and bottom-up triplet queries (BU-Qs) initialized from center-biased anchors; a self-attention mask preventing TD-to-BU information leakage; an Entity Pair Selector (EPS) that ranks entity pairs by content and spatial compatibility to limit TD-Q count; and the unified triplet decoder that jointly updates both query types so their outputs share a score scale without post-hoc calibration.

Load-bearing premise

The paper assumes that using an external object detector to split triplets into Det-T and UDet-T cleanly isolates the fundamental difference between detector-based and query-based reasoning. If the external detector's miss patterns differ systematically from those of the detectors inside the compared models, the observed complementarity could be partly an artifact of the specific detector chosen rather than a property of the two reasoning paradigms themselves.

What would settle it

Replace the external object detector used to define the Det-T/UDet-T split with a substantially different detector (different architecture, different miss patterns). If the complementarity between detector-based and query-based models disappears or reverses on the new split, the core empirical finding is detector-specific rather than paradigm-specific.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the complementarity finding generalizes, future SGG improvements should be evaluated on both Det-T and UDet-T subsets rather than aggregate metrics alone, since a model can improve aggregate recall while masking a collapse on one subset.
  • The dual-query principle — embedding two distinct reasoning priors in one decoder to avoid score-calibration problems — could transfer to other structured prediction tasks where DETR-style queries are used, such as human pose estimation or temporal action detection.
  • The center-biased initialization for BU-Qs suggests that spatial priors on query anchors matter for global exploration, which could inform anchor design in detection transformers more broadly.
  • As object detectors improve and UDet-T shrinks, the marginal benefit of BU-Qs should diminish; the paper's Swin-B experiment (UDet-T drops from 35% to 28%) confirms the benefit persists but is reduced, raising the question of when a detector-only approach would suffice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The complementarity may be partially an artifact of using a specific external detector to define the Det-T/UDet-T split. If a different detector with different miss patterns were used, the boundary between the two subsets would shift, potentially changing the magnitude of complementarity observed.
  • The paper does not report the overlap between what TD-Qs and BU-Qs predict when both succeed; a deeper analysis of whether the two query types converge on the same triplets or genuinely discover different ones would strengthen the complementarity claim.
  • The 14 FPS speed with 84.7M parameters is competitive, but the BU-Q count (800) is more than double the TD-Q count (300); the sensitivity of performance to this ratio is not explored, leaving open whether a more balanced allocation would change the trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper introduces a controlled experimental framework for analyzing the predictive behaviors of detector-based and query-based scene graph generation (SGG) methods from the perspective of 'detector-conditioned reachability.' Using an external object detector as a discriminator, the authors split ground-truth triplets into detected (Det-T) and undetected (UDet-T) subsets, revealing that detector-based methods excel on Det-T but collapse on UDet-T, while query-based methods show the complementary pattern. Motivated by this finding, the paper proposes Dual-SGG, which integrates both reasoning mechanisms within a single triplet decoder via a dual-query design: Top-Down Triplet Queries (TD-Qs) conditioned on detected entity pairs, and Bottom-Up Triplet Queries (BU-Qs) that globally explore triplets from center-biased initialization. The method is evaluated on Visual Genome, Open Images v6, and GQA-200, achieving state-of-the-art results while maintaining competitive inference speed (14.0 FPS).

Significance. The detector-conditioned reachability analysis (Tab. 1, Tab. 5) provides a useful diagnostic lens for understanding when and why detector-based versus query-based SGG methods fail, and the complementarity finding is well-motivated. The Dual-SGG architecture is a clean integration of both paradigms, and the ablation studies (Tab. 5–7) are thorough, including a strong-detector-condition analysis with Swin-B (Tab. 5, starred rows) and a complementarity comparison against post-hoc fusion baselines (Tab. 6). The method ships reproducible code and achieves SOTA on three datasets with competitive efficiency. The self-attention mask design to prevent TD-to-BU information leakage (Sec. 3.4, Tab. 7) is a thoughtful architectural detail. The paper provides falsifiable predictions: the micro-DR/micro-UDR decomposition quantitatively demonstrates that Dual-SGG maintains detector-based performance on Det-T while gaining on UDet-T.

major comments (4)
  1. Sec. 3, Eq. (1) and surrounding text: The paper motivates the unified decoder by arguing that directly applying max(S_d, S_q) to independent branches is problematic due to 'score scale inconsistencies' causing one branch to dominate. The proposed solution is joint training in a single decoder to eliminate the need for post-hoc calibration. However, at inference (Sec. 3.5), predictions from TD-Qs and BU-Qs are concatenated, NMS is applied, and TopK is selected. NMS only compares scores for spatially overlapping triplets, so it is inherently robust to global scale differences between branches — even if TD-Qs systematically produce higher scores than BU-Qs, NMS would still function correctly because it only needs to rank within overlapping pairs, not across the full set. This means the inference pipeline could work well even if the two branches are NOT well-calibrated relative to each other
  2. Sec. 3, Eq. (1) continued: The gap between Dual-SGG (33.5/38.5 R@K) and normalization fusion (32.2/37.3 R@K) in Tab. 6 is approximately 1.3 points, but this gap could arise from joint feature sharing or gradient flow benefits unrelated to score calibration per se. The paper does not analyze the actual score distributions of TD-Qs vs BU-Qs predictions to verify that joint training aligns their scales. Without such analysis, the specific claim that joint training eliminates the calibration problem identified in the motivation remains unverified. The authors should either (a) provide score distribution analysis showing that joint training does align TD-Q and BU-Q score scales, or (b) soften the motivation to acknowledge that joint training provides multiple benefits (feature sharing, gradient flow, reduced architectural complexity) beyond score calibration.
  3. Sec. 1 and Tab. 1: The detector-conditioned reachability split (Det-T vs UDet-T) is defined using an external object detector as a discriminator. The reader's weakest assumption concern is valid: if the external detector's recall characteristics differ significantly from the internal detectors used in the compared models (e.g., the detector-based model in Fig. 1a fine-tunes its own detector), the observed complementarity could partly reflect the specific detector choice rather than a fundamental property of the two reasoning paradigms. The paper does partially address this with the Swin-B experiment (Tab. 5, starred rows), showing robustness under a stronger detector. However, the external detector used for the Det-T/UDet-T split in Tab. 1 is the same Deformable-DETR that is then fine-tuned within the models, which creates a potential circularity: the discriminator is not fully external.
  4. Sec. 1 and Tab. 1 continued: The authors should clarify whether the Det-T/UDet-T split is computed on the pre-trained detector (before SGG fine-tuning) or after fine-tuning, and discuss whether the complementarity finding is robust to the choice of discriminator detector. This is load-bearing because the entire architectural motivation rests on the complementarity claim.
minor comments (8)
  1. Sec. 3.2, Eq. (6): The EPS uses different scoring at training (sigmoid(S_sp) × sigmoid(S_ct)) vs inference (sigmoid(S_ct)). The rationale for dropping the spatial score at inference is stated briefly but could benefit from a more explicit justification.
  2. Tab. 2: The #params for Dual-SGG is 84.7M, but the backbone (ResNet50) and detector parameters are not broken down. It would help readers to know how much of this is attributable to the triplet decoder vs the object detector.
  3. Sec. 3.3: The BU-Qs content embeddings are 'randomly initialized with a set of learnable vectors.' It would be useful to specify the dimensionality and whether these are shared across images or image-conditioned.
  4. Fig. 2: The EPS architecture diagram (a) is somewhat dense. Labeling the data flow more explicitly (e.g., marking which arrows correspond to S_sp vs S_ct) would improve readability.
  5. Sec. 4.3, Tab. 5: The baseline model (Fig. 1a) reports R@50 of 31.6, while in Tab. 1 the same model reports micro-R of 32.5. The difference is presumably due to Top-100 vs Top-50 evaluation, but this should be clarified.
  6. Sec. 3.4: The statement 'the predicate cost is excluded from the matching cost calculation, as its computational overhead outweighs its empirical benefit' could use a quantitative justification or reference.
  7. Tab. 6: The 'Average' fusion strategy performs dramatically worse (17.6 R@50) than all other methods. A brief explanation of why averaging fails so catastrophically would help readers understand the score scale issue.
  8. The paper references code availability ('Code is available at: Dual-SGG') but the actual URL is not provided in the text.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. The referee's comments center on two themes: (1) whether the motivation for joint training in a unified decoder—specifically the claim about eliminating score calibration problems—is fully supported by the evidence presented, and (2) whether the detector-conditioned reachability split (Det-T/UDet-T) is robust to the choice of external detector, given that the discriminator detector is the same architecture later fine-tuned within the models. We address both themes below. In brief, we agree that the score calibration claim should be softened and supplemented with empirical analysis, and we agree that the circularity concern regarding the discriminator detector warrants explicit clarification in the manuscript. We propose partial revisions addressing both points.

read point-by-point responses

  1. Referee: Sec. 3, Eq. (1) and surrounding text: The paper motivates the unified decoder by arguing that directly applying max(S_d, S_q) to independent branches is problematic due to 'score scale inconsistencies' causing one branch to dominate. The proposed solution is joint training in a single decoder to eliminate the need for post-hoc calibration. However, at inference (Sec. 3.5), predictions from TD-Qs and BU-Qs are concatenated, NMS is applied, and TopK is selected. NMS only compares scores for spatially overlapping triplets, so it is inherently robust to global scale differences between branches — even if TD-Qs systematically produce higher scores than BU-Qs, NMS would still function correctly because it only needs to rank within overlapping pairs, not across the full set. This means the inference pipeline could work well even if the two branches are NOT well-calibrated relative to each other

    Authors: The referee raises a valid and technically precise point. We agree that NMS, as applied in our inference pipeline, operates on spatially overlapping triplets and is therefore robust to global scale differences between the TD-Q and BU-Q branches. The referee is correct that the inference pipeline could function adequately even without perfect inter-branch calibration, because NMS only needs to rank within overlapping pairs rather than across the full candidate set. Our original motivation overstates the necessity of joint training for score calibration at inference time. We will revise the manuscript to acknowledge this explicitly. That said, we note that the calibration concern is not entirely vacuous: the final TopK selection step does rank across the full candidate pool (after NMS deduplication), so systematic scale differences could still influence which triplets survive into the final output. However, this is a weaker claim than what we originally stated, and we will adjust the text accordingly. revision: partial

  2. Referee: Sec. 3, Eq. (1) continued: The gap between Dual-SGG (33.5/38.5 R@K) and normalization fusion (32.2/37.3 R@K) in Tab. 6 is approximately 1.3 points, but this gap could arise from joint feature sharing or gradient flow benefits unrelated to score calibration per se. The paper does not analyze the actual score distributions of TD-Qs vs BU-Qs predictions to verify that joint training aligns their scales. Without such analysis, the specific claim that joint training eliminates the calibration problem identified in the motivation remains unverified. The authors should either (a) provide score distribution analysis showing that joint training does align TD-Q and BU-Q score scales, or (b) soften the motivation to acknowledge that joint training provides multiple benefits (feature sharing, gradient flow, reduced architectural complexity) beyond score calibration.

    Authors: We accept this criticism. The 1.3-point gap between Dual-SGG and normalization fusion in Tab. 6 could indeed arise from multiple sources—joint feature sharing, gradient flow, shared decoder representations—rather than from score scale alignment alone. We did not provide score distribution analysis for TD-Qs vs. BU-Qs, and without that analysis, the specific claim that joint training eliminates the calibration problem is not empirically verified. We will adopt option (b): we will soften the motivation in Sec. 3 to acknowledge that joint training in the unified decoder provides multiple benefits beyond score calibration, including shared feature representations, gradient flow between the two reasoning pathways, and reduced architectural complexity. We will also add the requested score distribution analysis (histograms of TD-Q vs. BU-Q triplet scores on the VG validation set) in the supplementary material, so that readers can assess the degree to which joint training aligns the score scales. This will make the motivation more precise and the evidence more complete. revision: partial

  3. Referee: Sec. 1 and Tab. 1: The detector-conditioned reachability split (Det-T vs UDet-T) is defined using an external object detector as a discriminator. The reader's weakest assumption concern is valid: if the external detector's recall characteristics differ significantly from the internal detectors used in the compared models (e.g., the detector-based model in Fig. 1a fine-tunes its own detector), the observed complementarity could partly reflect the specific detector choice rather than a fundamental property of the two reasoning paradigms. The paper does partially address this with the Swin-B experiment (Tab. 5, starred rows), showing robustness under a stronger detector. However, the external detector used for the Det-T/UDet-T split in Tab. 1 is the same Deformable-DETR that is then fine-tuned within the models, which creates a potential circularity: the discriminator is not fully external.

    Authors: We appreciate this careful observation. To clarify: the Det-T/UDet-T split in Tab. 1 is computed using the pre-trained Deformable-DETR before SGG fine-tuning—that is, the discriminator is the frozen, pre-trained detector, not the fine-tuned version. We will state this explicitly in the revised manuscript. That said, the referee's circularity concern is legitimate: even though the split is computed before fine-tuning, the discriminator is the same architecture (Deformable-DETR with ResNet-50) that is subsequently fine-tuned within the models, so the discriminator is not fully independent of the models being evaluated. The Swin-B experiment (Tab. 5, starred rows) partially addresses this by showing that the complementarity finding holds under a different, stronger detector configuration, but we agree this does not fully eliminate the concern since Swin-B is still a Deformable-DETR variant. We will add a discussion of this limitation in the revised manuscript, acknowledging that the complementarity finding is conditioned on the detector family used and that a fully external discriminator (e.g., a different detector architecture entirely) would provide stronger evidence. We believe the complementarity is a fundamental property of the two reasoning paradigms—detector-based methods are architecturally constrained to triplets whose entities are detected, while query-based methods are not—but we agree that the current experimental design does not fully rule out the influence of the specific detector choice. revision: partial

  4. Referee: Sec. 1 and Tab. 1 continued: The authors should clarify whether the Det-T/UDet-T split is computed on the pre-trained detector (before SGG fine-tuning) or after fine-tuning, and discuss whether the complementarity finding is robust to the choice of discriminator detector. This is load-bearing because the entire architectural motivation rests on the complementarity claim.

    Authors: As noted in our response above, the Det-T/UDet-T split is computed on the pre-trained detector before SGG fine-tuning. We will add this clarification to Sec. 1 of the revised manuscript. Regarding robustness to the choice of discriminator: the Swin-B experiment (Tab. 5, starred rows) provides partial evidence, as it uses a different backbone and more entity queries, yet the complementarity pattern persists—Dual-SGG w/o BU-Qs maintains high micro-DR but low micro-UDR, while Dual-SGG w/o TD-Qs shows the complementary pattern. However, we acknowledge that both the default and Swin-B configurations are Deformable-DETR variants, so the evidence is limited to one detector family. We will add an explicit discussion of this limitation and note that validating with a structurally different detector architecture (e.g., a Faster R-CNN-based or DINO-based discriminator) would strengthen the generality claim. We consider this an important direction for future work but note that the core architectural argument—that detector-based reasoning is constrained by detection reachability while query-based reasoning is not—holds by construction, independent of the specific detector used to demonstrate it. revision: partial

Circularity Check

0 steps flagged

No significant circularity; one minor self-citation used as comparison baseline, not as load-bearing premise.

full rationale

The paper's derivation chain is empirical and self-contained. (1) The pre-ablation study (Table 1) establishes complementarity between detector-based and query-based models using an external detector to define the Det-T/UDet-T split — this is an independent empirical observation, not a definitional consequence. (2) The method (Dual-SGG) is a new architecture with dual queries (TD-Qs, BU-Qs) trained jointly; its performance on VG, OIv6, and GQA-200 test sets is evaluated against external benchmarks, not against fitted values. (3) The complementarity analysis (Table 6) compares joint training against post-hoc fusion strategies (max, average, normalization) — this is a genuine ablation, not a result forced by construction. (4) The self-citation [34] (Salience-SGG, Qu et al.) shares authors with the present paper but is used solely as a comparison baseline in Tables 2-4, not as a foundational axiom or uniqueness theorem that the paper's argument depends on. No equation reduces to its inputs by definition, and no 'prediction' is a renamed fit. The skeptic's concern about NMS masking scale mismatch is a correctness risk, not a circularity issue. Overall, the central claims have independent empirical content.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 3 invented entities

The free parameters are standard hyperparameters for DETR-based models. The axioms are domain assumptions standard in the SGG literature. The invented entities (TD-Qs, BU-Qs, EPS) are architectural components evaluated on external benchmarks with ablation studies.

free parameters (6)
  • N_e (entity queries) = 200
    Set for the object detector; standard for Deformable-DETR but a chosen capacity.
  • N_td (top-down triplet queries) = 300
    Number of entity pairs selected by EPS for TD-Qs; chosen hyperparameter.
  • N_bu (bottom-up triplet queries) = 800
    Number of BU-Qs for global prediction; chosen hyperparameter.
  • L (decoder layers) = 4
    Number of transformer blocks in the triplet decoder.
  • lambda_cls, lambda_box, lambda_giou = 2, 5, 2
    Loss weights for object detector; standard values from prior work.
  • tau (logit adjustment) = 0.2, 0.3
    Debiasing parameter for Unbiased-SGG experiments; selected for best results.
axioms (3)
  • domain assumption DETR-based query-based SGG methods can predict triplets without enumerating entity pairs.
    Foundational assumption from prior work [1, 15] that query-based methods can bypass the detector constraint. Invoked in Sec. 1 and Sec. 3.
  • domain assumption An external object detector can serve as a discriminator for detector-conditioned reachability.
    The reachability analysis (Sec. 1) depends on the assumption that the external detector's detection success/failure defines the 'reachable' space for detector-based SGG models.
  • domain assumption NMS serves as a practical implementation of the max operation in Eq. (1).
    Stated in Sec. 3.5 Inference. This assumes NMS on concatenated predictions from a single decoder approximates the theoretical max of two score functions.
invented entities (3)
  • Top-Down Triplet Queries (TD-Qs) independent evidence
    purpose: Detector-conditioned reasoning within the triplet decoder
    Evaluated on three public benchmarks with ablation studies isolating their contribution (Tab. 5, 6).
  • Bottom-Up Triplet Queries (BU-Qs) independent evidence
    purpose: Global image-conditioned prediction to capture triplets unreachable by TD-Qs
    Evaluated on three public benchmarks with ablation studies isolating their contribution (Tab. 5, 6).
  • Entity Pair Selector (EPS) independent evidence
    purpose: Selects entity pairs for TD-Q initialization to reduce computational cost
    Ablation study in Tab. 7 shows the effect of its components (Ssp, Sct).

reviewed 2026-07-08 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability." pith.science (2026). https://pith.science/paper/2CVY6D4W

@misc{pith2026260706176,
  author       = {Pith},
  title        = {Pith review of: Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CVY6D4W}},
  note         = {Machine review of arXiv:2607.06176}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Scene graph generation (SGG) approaches can be broadly classified into detector-based and query-based methods according to their underlying reasoning mechanisms. However, the discrepancy in their predictive behaviors, induced by these distinct mechanisms, has not been systematically analyzed. In this work, we design a controlled experimental setup to examine prediction discrepancies from the perspective of detector-conditioned reachability. The results suggest clear complementary clues. Motivated by this observation, we introduce a Dual-SGG method that consolidates both reasoning mechanisms via a dual-query design, thereby leveraging the complementary predictive behaviors of both detector-based and query-based methods. Extensive experiments on the Visual Genome, Open Images v6, and GQA-200 datasets demonstrate the effectiveness of the proposed method.

Figures

Figures reproduced from arXiv: 2607.06176 by Julie Ouerfelli-Ethier, Klaus Obermayer, Olaf Hellwich, Ole Hall, Pia K Bideau, Runfeng Qu.

Figure 1
Figure 1. Figure 1: SGG Models. (a) Detector-based model. (b) Query-based model. (c) Query￾based model that additionally incorporates entity information derived from an object detector. (d) The proposed Dual-SGG method, which integrates detector-based and query-based reasoning mechanisms within a unified triplet decoder. spatially uncovered by detections from the employed object detector. Here, we term this problem the detect… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of our Dual-SGG method. It includes an object detector, an entity pairs selector (EPS), and a triplet decoder. FFN denotes the feed-forward network. TD content and TD anchors form TD-Qs, while BU content and BU anchors form BU-Qs. (a) presents architectural details of EPS. set of entity candidates, denoted as E = {ei} = {(q e i , be i , ce i )} Ne i=1, where q e i ∈ R d , b e i ∈ R 4 , and c e… view at source ↗
Figure 3
Figure 3. Figure 3: Attention visualization. Columns 2–5 and 6-end depict the attention distri￾butions of BU-Qs and TD-Qs across the decoder layers (li), respectively. man-wearing-hat person-wearing-boot man-holding-skateboard man-wearing-hat person-wearing-boot man-wearing-hat person-wearing-boot man-holding-skateboard man hat skateboard person boot (a) boy1-in-boat1 boy2-in-boat1 boy1-in-boat1 boy3-in-boat2 boy1-in-boat1 bo… view at source ↗
Figure 4
Figure 4. Figure 4: Example visualization. For each image, the detected ground-truth triplets by Dual-SGG w/o TD-Qs (top), Dual-SGG w/o BU-Qs (middle), and Dual-SGG (bot￾tom), are provided. Triplets highlighted in bold denote missed triplets by at least one of the other two models. 5 Conclusion This study investigates the distinct predictive behaviors of detector-based and query-based SGG methods from the perspective of detec… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 52 canonical work pages · 6 internal anchors

  1. [1]

    In: European conference on computer vision

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)

    work page 2020

  2. [2]

    Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation

    Chen, M., Chen, G., Wang, W., Yang, Y.: Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. arXiv preprint arXiv:2409.10262 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    In: Proceedings of the IEEE/CVF international conference on com- puter vision

    Chen, Q., Chen, X., Wang, J., Zhang, S., Yao, K., Feng, H., Han, J., Ding, E., Zeng, G., Wang, J.: Group detr: Fast detr training with group-wise one-to-many assignment. In: Proceedings of the IEEE/CVF international conference on com- puter vision. pp. 6633–6642 (2023)

    work page 2023

  4. [4]

    In: European Conference on Computer Vision

    Chen, Y.T., Shi, J., Ye, Z., Mertz, C., Ramanan, D., Kong, S.: Multimodal ob- ject detection via probabilistic ensembling. In: European Conference on Computer Vision. pp. 139–158. Springer (2022)

    work page 2022

  5. [5]

    IEEE Transactions on Pattern Analysis and Machine Intelli- gence45(9), 11169–11183 (2023)

    Cong, Y., Yang, M.Y., Rosenhahn, B.: Reltr: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelli- gence45(9), 11169–11183 (2023)

    work page 2023

  6. [6]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Desai, A., Wu, T.Y., Tripathi, S., Vasconcelos, N.: Learning of visual relations: The devil is in the tails. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15404–15413 (2021)

    work page 2021

  7. [7]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Dong, X., Gan, T., Song, X., Wu, J., Cheng, Y., Nie, L.: Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19427–19436 (2022)

    work page 2022

  8. [8]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, J., Zhang, T., Chen, K., Dou, Q.: Hybrid reciprocal transformer with triplet feature alignment for scene graph generation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8953–8963 (2025)

    work page 2025

  9. [9]

    In: European conference on computer vision

    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European conference on computer vision. pp. 630–645. Springer (2016)

    work page 2016

  10. [10]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    work page 2019

  11. [11]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Im, J., Nam, J., Park, N., Lee, H., Park, S.: Egtr: Extracting graph from trans- former for scene graph generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24229–24238 (2024)

    work page 2024

  12. [12]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Jia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L., Zhang, C., Hu, H.: Detrs with hybrid matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19702–19712 (2023)

    work page 2023

  13. [13]

    In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition

    Johnson, J., Gupta, A., Fei-Fei, L.: Image generation from scene graphs. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 1219–1228 (2018)

    work page 2018

  14. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jung, D., Kim, S., Kim, W.H., Cho, M.: Devil’s on the edges: Selective quad attention for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18664–18674 (2023)

    work page 2023

  15. [15]

    Advances in Neural Information Processing Systems35, 24295–24308 (2022)

    Khandelwal, S., Sigal, L.: Iterative scene graph generation. Advances in Neural Information Processing Systems35, 24295–24308 (2022)

    work page 2022

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kim, J., Park, J., Park, J., Kim, J., Kim, S., Kim, H.J.: Groupwise query specializa- tion and quality-aware multi-assignment for transformer-based visual relationship detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 28160–28169 (2024) Dual-SGG 17

    work page 2024

  17. [17]

    International journal of computer vision123(1), 32–73 (2017)

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision123(1), 32–73 (2017)

    work page 2017

  18. [18]

    Naval research logistics quarterly2(1-2), 83–97 (1955)

    Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2), 83–97 (1955)

    work page 1955

  19. [19]

    International journal of computer vision128, 1956–1981 (2020)

    Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Ka- mali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision128, 1956–1981 (2020)

    work page 1956

  20. [20]

    In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition

    Li, F., Zhang, H., Liu, S., Guo, J., Ni, L.M., Zhang, L.: Dn-detr: Accelerate detr training by introducing query denoising. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 13619–13627 (2022)

    work page 2022

  21. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, J., Wang, Y., Guo, X., Yang, R., Li, W.: Leveraging predicate and triplet learning for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 28369–28379 (2024)

    work page 2024

  22. [22]

    In: proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Li, R., Zhang, S., He, X.: Sgtr: End-to-end scene graph generation with trans- former. In: proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 19486–19496 (2022)

    work page 2022

  23. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, R., Zhang, S., Wan, B., He, X.: Bipartite graph network with adaptive message passing for unbiased scene graph generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11109–11119 (2021)

    work page 2021

  24. [24]

    Advances in Neural Information Pro- cessing Systems32(2019)

    Li, Y., Ma, T., Bai, Y., Duan, N., Wei, S., Wang, X.: Pastegan: A semi-parametric method to generate image from scene graph. Advances in Neural Information Pro- cessing Systems32(2019)

    work page 2019

  25. [25]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)

    work page 2017

  26. [26]

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection.In:ProceedingsoftheIEEEinternationalconferenceoncomputervision. pp. 2980–2988 (2017)

    work page 2017

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lin, X., Ding, C., Zhang, J., Zhan, Y., Tao, D.: Ru-net: Regularized unrolling network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19457–19466 (2022)

    work page 2022

  28. [28]

    DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

    Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., Zhang, L.: Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  29. [29]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchicalvisiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)

    work page 2021

  30. [30]

    Decoupled Weight Decay Regularization

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  31. [31]

    Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with languagepriors.In:Europeanconferenceoncomputervision.pp.852–869.Springer (2016)

    work page 2016

  32. [32]

    Long-tail learning via logit adjustment

    Menon, A.K., Jayasumana, S., Rawat, A.S., Jain, H., Veit, A., Kumar, S.: Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2007

  33. [33]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV)

    Nguyen, K., Tripathi, S., Du, B., Guha, T., Nguyen, T.Q.: In defense of scene graphs for image captioning. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV). pp. 1407–1416 (October 2021) 18 R. Qu et al

    work page 2021

  34. [34]

    arXiv preprint arXiv:2601.08728 (2026)

    Qu, R., Hall, O., Bideau, P.K., Ouerfelli-Ethier, J., Rolfs, M., Obermayer, K., Hell- wich, O.: Salience-sgg: Enhancing unbiased scene graph generation with iterative salience estimation. arXiv preprint arXiv:2601.08728 (2026)

    work page arXiv 2026

  35. [35]

    IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object de- tection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence39(6), 1137–1149 (2016)

    work page 2016

  36. [36]

    In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Shi, J., Zhang, H., Li, J.: Explainable and explicit visual reasoning over scene graphs. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 8376–8384 (2019)

    work page 2019

  37. [37]

    In: European conference on computer vision

    Shit, S., Koner, R., Wittmann, B., Paetzold, J., Ezhov, I., Li, H., Pan, J., Shar- ifzadeh, S., Kaissis, G., Tresp, V., et al.: Relationformer: A unified framework for image-to-graph generation. In: European conference on computer vision. pp. 422–439. Springer (2022)

    work page 2022

  38. [38]

    In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision

    Sudhakaran, G., Dhami, D.S., Kersting, K., Roth, S.: Vision relation transformer for unbiased scene graph generation. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. pp. 21882–21893 (2023)

    work page 2023

  39. [39]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Tang, K., Zhang, H., Wu, B., Luo, W., Liu, W.: Learning to compose dynamic tree structures for visual contexts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6619–6628 (2019)

    work page 2019

  40. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Teng, Y., Wang, L.: Structured sparse r-cnn for direct scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19437–19446 (2022)

    work page 2022

  41. [41]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    work page 2017

  42. [42]

    In: European Conference on Computer Vision

    Wang, L., Yuan, Z., Chen, B.: Multi-granularity sparse relationship matrix predic- tion network for end-to-end scene graph generation. In: European Conference on Computer Vision. pp. 105–121. Springer (2024)

    work page 2024

  43. [43]

    Journal of Visual Communication and Image Representation58, 477–485 (2019)

    Xu, N., Liu, A.A., Liu, J., Nie, W., Su, Y.: Scene graph captioner: Image captioning based on structural visual representation. Journal of Visual Communication and Image Representation58, 477–485 (2019)

    work page 2019

  44. [44]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Yoon, K., Kim, K., Jeon, J., In, Y., Kim, D., Park, C.: Ra-sgg: retrieval-augmented scene graph generation framework via multi-prototype learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 9562–9570 (2025)

    work page 2025

  45. [45]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural motifs: Scene graph parsing with global context. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5831–5840 (2018)

    work page 2018

  46. [46]

    In: European conference on computer vision

    Zhang, A., Yao, Y., Chen, Q., Ji, W., Liu, Z., Sun, M., Chua, T.S.: Fine-grained scene graph generation with data transfer. In: European conference on computer vision. pp. 409–424. Springer (2022)

    work page 2022

  47. [47]

    detr: Instructive multi-route training for de- tection transformers

    Zhang, C.B., Zhong, Y., Han, K.: Mr. detr: Instructive multi-route training for de- tection transformers. In: Proceedings of the Computer Vision and Pattern Recog- nition Conference. pp. 9933–9943 (2025)

    work page 2025

  48. [48]

    In: Proceedings of the IEEE conference on com- puter vision and pattern recognition

    Zhang, H., Kyaw, Z., Chang, S.F., Chua, T.S.: Visual translation embedding net- work for visual relation detection. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition. pp. 5532–5540 (2017)

    work page 2017

  49. [49]

    DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

    Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.Y.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 (2022) Dual-SGG 19

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  50. [50]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhao, C., Sun, Y., Wang, W., Chen, Q., Ding, E., Yang, Y., Wang, J.: Ms-detr: Efficient detr training with mixed supervision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17027–17036 (2024)

    work page 2024

  51. [51]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zheng, C., Lyu, X., Gao, L., Dai, B., Song, J.: Prototype-based embedding net- work for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22783–22792 (2023)

    work page 2023

  52. [52]

    Deformable DETR: Deformable Transformers for End-to-End Object Detection

    Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

This paper was first reviewed by glm-5.2 on July 8, 2026.