{"id":"677c09b6-cd8b-4c15-8bfb-1591493e8e5d","arxiv_id":"2411.13287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TA-HDG combines heterogeneous and dual graph message passing with interactive/non-interactive relation typing to improve both head and tail class scene graph prediction.","lead":"This paper introduces TA-HDG, a scene graph generation model that combines heterogeneous and dual graphs and uses type-aware message passing to improve prediction of rare relations while keeping common relations strong. The authors report state-of-the-art results on Visual Genome and Open Images benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The interactive/non-interactive split is both underspecified and, as described, cannot rebalance the long tail: Fig. 7's ratio change is renormalization and Sec. III-C applies plain BCE, so the type-aware mechanism behind the SOTA claim is unverified.","rationale":"The tables and ablations provide genuine support for a graph-based architecture: Table II shows monotonic gains from adding HDGC and TAMP on VG, Table V repeats this on OI, and the method is evaluated under standard metrics and settings. I found no internal numerical contradiction that would falsify the overall SOTA claim. The soft spot is the type-categorization mechanism, which the paper itself lists as its first contribution. The reader identified the missing mapping; I would go further and note that supplying a mapping alone would not rescue the stated 'distribution adjustment' story, because plain BCE training (Eq. 14) and renormalized conditional frequencies (Fig. 7/Fig. 8) do not rebalance the long-tailed training distribution. A random-split control would settle whether the super-type taxonomy matters. Until that control is run, or until the mapping and any type-dependent loss are supplied, the central causal claim is unverified, so the reader's CONDITIONAL verdict remains appropriate; the paper should not be rejected on the available evidence, but it also should not be accepted as-is.","tokens_in":18409,"tokens_out":8669,"duration_ms":98028,"concrete_test":"Run the VG SGDet experiment with gamma replaced by a random balanced binary partition of the 50 predicates (five seeds), keeping all other hyperparameters fixed. If R@100/mR@100 remain within noise of Table I (33.71 / 38.21 / 14.85 / 16.66), the specific Motifs-derived taxonomy is not load-bearing. Additionally, instrument the released code to verify whether any type-dependent loss weight or sampling schedule exists; Sec. III-C indicates none, so if none is found, the 'distribution adjustment' claims in the Effectiveness of the relation categorization subsections should be revised or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical claim in Sec. V rests on the type-aware mechanism: TA-HDG improves head/tail performance because relations are first split into interactive and non-interactive types and message passing is conditioned on those types. Two problems compromise this mechanism. (1) The split is never specified. Sec. III-A invokes 'relation super-types in Motifs' and then defines gamma with 'the Mean (·) is utilized as the pre-defined function gamma,' but gamma: C -> T must assign each of the 50/31 predicates to phi or delta; no such table or formula is given, so Eq. (7) and Eq. (9), which use type-specific weight matrices W^i_t and W^j_t, cannot be reproduced or audited. (2) Even if the mapping were supplied, the described training procedure does not perform any distribution rebalancing: Sec. III-C minimizes plain BCE losses (Eq. 14) with no per-type weighting or sampling, and the evidence in Fig. 7 is a renormalization artifact — 'hanging from' rises from 0.0023 to 0.0272 only because the denominator changes to the type total. Conditional probabilities within a type do not alter the training-set frequency of any predicate. Thus the causal story 'categorizing relation types alleviates the long-tail' is unsupported, and the reported mR@K gains cannot be attributed to this component as opposed to the graph construction and subject-object pair selection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TA-HDG, an unbiased scene graph generation (SGG) framework that combines a heterogeneous graph and a dual graph to model interactions between objects and between relations. It introduces a subject-object pair selection strategy based on distance, confidence, and existence information, and a Type-Aware Message Passing (TAMP) module with Intra-Type and Inter-Type stages. The authors claim state-of-the-art results on Visual Genome and Open Images, improving both R@K and mR@K across SGDet, SGCls, and PredCls, and attribute the gains to categorizing relations into interactive and non-interactive types.","tokens_in":18812,"tokens_out":3041,"duration_ms":31971,"significance":"If the empirical claims hold, the paper would contribute a competitive architecture for unbiased SGG, with a useful combination of heterogeneous and dual graphs and a sensible subject-object pair selection strategy. The ablations in Tables II and V are systematic and show consistent improvements as components are added, which is a strength. However, the central mechanistic claim—that type-aware message passing driven by an interactive/non-interactive relation split alleviates the long-tail problem—is not currently verifiable because the relation type mapping is undefined, and the provided evidence (Fig. 7) is a renormalization artifact. The significance is therefore conditional on the authors supplying the missing specification and either implementing actual rebalancing or revising the attribution.","major_comments":[{"comment":"The relation type mapping γ is never defined. The text states 'Note that the Mean (·) is utilized as the pre-defined function γ' (Sec. III-A), but γ : C → T must assign each of the 50 VG or 31 OI relation classes to either φ (interactive) or δ (non-interactive); 'Mean' is an aggregation operator, not a class-to-type assignment. No table or formula lists which predicates belong to which type, and the reference to 'relation super-types in Motifs [27]' does not supply this information because Motifs does not define such a binary split. Since Eqs. (7) and (9) use type-specific weight matrices W^i_t, W^j_t, W^{i→j}_t, the core type-aware mechanism is not reproducible or auditable. The authors must provide the explicit mapping.","section":"Sec. III-A, Eqs. (7) and (9)"},{"comment":"The existence matrix M_l is not specified. The paper says it is 'calculated based on the class labels and co-linearities' but gives no formula, definition of co-linearity, or reference. Along with the undefined thresholds s_b and s_l, this makes the subject-object pair selection strategy irreproducible. This matters because Tables II and V attribute part of the performance gain to this selection strategy.","section":"Sec. III-A, Eq. (3)"},{"comment":"The claim that categorizing relations into interactive and non-interactive types 'adjusts the distribution of relations' and alleviates the long-tail problem is not supported. The training loss in Eq. (14) is a plain BCE loss with no per-type weighting, sampling, or any other form of rebalancing. Fig. 7 shows the proportion of a relation within its assigned type (e.g., 'hanging from' rising from 0.0023 to 0.0272), which is a renormalization artifact: conditional shares within a type do not change the training-set frequency of any predicate. Consequently, the mR@K improvements in Tables I and IV cannot be attributed to the type categorization mechanism as opposed to the graph construction and subject-object pair selection. The authors should either introduce explicit per-type rebalancing in the loss or training procedure, or revise the causal narrative to match what the method actually does.","section":"Sec. III-C and Fig. 7"},{"comment":"No error bars or statistical significance tests are reported. Several of the claimed improvements are very small (e.g., VG SGDet R@50: TA-HDG 33.71 vs. HL-Net 33.70, a 0.01 difference), and all comparisons are single runs. Without variance estimates or repeated trials, the state-of-the-art claim is fragile, especially for the metrics where the gap is within the expected run-to-run variability of such models.","section":"Tables I and IV"}],"minor_comments":[{"comment":"There is a typo: 'inter-relaitons' should be 'inter-relations'.","section":"Abstract"},{"comment":"The text says the attention score α^{d,(l)}_{i→j} is 'calculated in the same way following Eq. 6', but the analogous attention formula is Eq. (5); this cross-reference should be corrected.","section":"Sec. III-B, Eq. (6)"},{"comment":"The acronym 'NSM' in Sec. IV-A should be 'NMS' (non-maximum suppression). Also, Table V has a typo in the column header 'wmAPpht', which should be 'wmAPphr'.","section":"Sec. IV-A and Table V"},{"comment":"The method name 'USGG-HD' in the figure labels appears to be a leftover from an earlier version; it should be 'TA-HDG'.","section":"Figs. 9 and 10"},{"comment":"The initial feature notation is confusing: the text says the initial representations are f^{h,(0)}_{i→j} = f^{d,(L+1)}_{i→j} and f^{h,(0)}_i = f^{d,(L+1)}_i, but the dual-graph superscript uses (l+1) while the layer index for the heterogeneous graph is also (l); please clarify the layer indexing and the transition from the dual graph to the heterogeneous graph.","section":"Sec. III-B, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a typical empirical SGG submission. The main concern is not that the architecture is wrong but that the type-aware mechanism, which is central to the paper's narrative, is underspecified and the provided evidence for its effect is not valid as stated. The authors should be asked to specify the relation type mapping, define the existence/co-linearity matrix, and either implement actual rebalancing or reframe the contribution without claiming that the type split rebalances the distribution. If these points are addressed, the paper may be publishable, but as it stands the central claim is not fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core take: this is an incremental but plausible scene graph generation paper. The new bit is real: combining a heterogeneous graph and a dual graph, then running separate intra-type (objects-objects, relations-relations) and inter-type (objects-relations, relations-objects) message passing with type-conditioned weights. That combination is not in HetSGG or EdgeSGG, and the ablations on both VG and OI are internally consistent: each added component improves R@K and mR@K, and the subject-object pair selection study is reasonably thorough. The paper earns credit for that.\n\nThe soft spots are not fatal to the architecture, but they undercut the stated mechanism. First, the interactive/non-interactive split is never specified. The text says the pre-defined function gamma uses the Mean(·), which is not a mapping from the 50 VG or 31 OI predicates to the two types. Without that mapping, Equations (7) and (9) cannot be reproduced or audited. This is the biggest concrete gap.\n\nSecond, the stress-test is right about the distribution story. Figure 7's ratio change from 0.0023 to 0.0272 is just renormalization: conditional frequency within a type goes up while the global training frequency of the predicate does not change. And the training loss in Eq. (14) is plain BCE with no per-type weighting or sampling. So the claim that categorizing relation types adjusts the distribution and thereby alleviates the long tail is not supported by the described procedure. The mR@K gains might come from the graph construction, the pair selection, or the type-conditioned message passing, but not from distribution rebalancing as described.\n\nThe absence of code and error bars is a third concern, especially since comparisons rely on published numbers under potentially different settings. None of this makes the empirical claim self-contradictory; the ablations are consistent and the gains are plausible. But the burden is on the authors to specify the type mapping and either show a real rebalancing step or drop that part of the narrative.\n\nWho is this for? Researchers working on SGG graph construction and long-tail relation prediction will find the architecture worth reading. It deserves a serious referee, not a desk reject. I would ask for the type mapping, code/data release, and error bars before accepting; without those, the central mechanism remains unverified.\n\nRecommendation: send to peer review with a request for major revision and reproducibility details.","headline":"A workmanlike SGG architecture paper with a genuinely new graph combination and consistent benchmark gains, but the advertised long-tail rebalancing mechanism is not actually implemented as described and the type split is underspecified.","tokens_in":724,"tokens_out":1811,"would_cite":false,"duration_ms":42090,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph network that splits relations into two types improves rare and common scene graph prediction.","keywords":["scene graph generation","unbiased scene graph generation","long-tail relation prediction","heterogeneous graph","dual graph","message passing","Visual Genome","Open Images"],"falsifier":"Run the same TA-HDG pipeline with the relation-type assignment replaced by a random partition of the 50 Visual Genome predicates into two groups, preserving all other settings. If the mean-recall gains on SGCls remain close to the reported values, the type-aware split is not the operative mechanism; if they collapse, the split is load-bearing.","tokens_in":18234,"feed_emoji":"🖼️","tokens_out":8397,"duration_ms":77272,"temperature":0.7,"pith_summary":"The paper aims to show that the long-tail problem in scene graph generation can be addressed by modeling relations as two complementary graph views and by separating relation classes into interactive and non-interactive types before passing messages. It introduces TA-HDG, which builds a heterogeneous graph with objects as nodes and typed relations as edges, plus a dual graph with relations as nodes and shared objects as edges. On the dual graph it passes intra-type messages among relations and among objects; on the heterogeneous graph it passes inter-type messages between objects and relations separately for the two relation types. Experiments on Visual Genome and Open Images report higher recall and mean recall than the compared methods on all three subtasks, which the paper interprets as evidence that tail relations improve without sacrificing head relations.","feed_headline":"Type-split message passing lifts rare and common scene relations","feed_subtitle":"Separating relations into interactive and non-interactive types lifts recall and mean recall on both datasets.","key_machinery":"The load-bearing mechanism is the combination of Heterogeneous and Dual Graph Construction (HDGC) with Type-Aware Message Passing (TAMP). HDGC starts from a fully connected object graph, prunes edges with a subject-object pair selection strategy that uses distance, confidence, and existence matrices, then builds a heterogeneous graph (objects as nodes, two types of relation edges) and a dual graph (relations as nodes, shared objects as edges). TAMP then refines features in two stages: the Intra-Type stage passes messages among relations and among objects on the dual graph, capturing context within the same type, and the Inter-Type stage passes messages between objects and relations on the heterogeneous graph, using separate weight matrices for interactive and non-interactive relations. The pre-defined function $\\gamma$ maps relation classes to the two types, and the paper states that the Mean(·) is used as this function.","core_discovery":"TA-HDG's central claim is that simultaneously modeling interactions among relations and interactions among objects—not just object-relation interactions—lets a scene graph generator refine features well enough to predict rare relations while keeping common relations accurate. The paper argues that prior methods neglect the 'intra-type' semantic context among different relations sharing an object and among different objects sharing a relation, and that this omission is why they favor head classes or lose them when rebalancing. To fix this, TA-HDG constructs a heterogeneous graph to model the dependence of relations on objects and a dual graph to model interactions between relations, then applies Type-Aware Message Passing in two stages: Intra-Type message passing on the dual graph and Inter-Type message passing on the heterogeneous graph, with the relation type (interactive or non-interactive) controlling the weight matrices. The reported result is that this architecture improves R@K and mR@K on Visual Genome and Open Images in all three evaluation modes, with ablations attributing the gains to both the dual-graph stage and the heterogeneous-graph stage.","pith_inferences":["A natural extension the paper does not run is a learned type assignment: if the interactive/non-interactive split is the true source of the rebalancing, a network that infers the type from data should match or beat the fixed Motifs-based split; the current paper only compares against the fixed split.","Because the paper's own conclusion names detector dependence as a limit, a testable next step is to attach HDGC and TAMP to a detector-free panoptic scene graph architecture to see whether the type-aware context transfer survives the change.","The dual graph's computational cost is acknowledged as high; an unstated consequence is that approximating the shared-object edges, for example by sampling relation neighborhoods, would determine how much of the gain is due to full relation-relation connectivity versus the type-aware message-passing design.","The binary type split could be seen as a simple prior that might transfer to other long-tailed relational prediction tasks, such as human-object interaction detection; this is an inference from the paper's mechanism, not something it tests."],"forward_implications":["On Visual Genome, TA-HDG reports top R@100 and mR@100 values in SGDet, SGCls, and PredCls, with mR@100 gains of 7.46, 9.63, and 11.39 percentage points over HL-Net in the three tasks.","On Open Images, TA-HDG reports the best results on the weighted composite score (scorewtd = 45.19) by improving both R@50 (81.71) and mR@50 (43.28) over the compared methods.","Ablations show that removing the heterogeneous graph + Inter-Type message passing lowers mR@K by at least 2% on SGDet, while removing the dual graph + Intra-Type message passing lowers R@K by about 1.5%; the paper reads this as the two stages contributing to tail and head relations respectively.","The subject-object pair selection strategy, combining distance, confidence, and existence information, improves pair recall (pR@100) by roughly 2% over confidence-only selection on Visual Genome, and transfers to Open Images without retuning.","Relation categorization raises the effective distribution ratio of each relation (for example, hanging from from 0.0023 to 0.0272 on Visual Genome), which the paper argues reduces data bias."],"supporting_citations":[{"why":"Supplies the relation super-types, referenced as the basis for categorizing relations into interactive and non-interactive types.","marker":"[27]"},{"why":"HetSGG is the heterogeneous graph approach that HDGC extends, and it is the main previous baseline on mean recall.","marker":"[8]"},{"why":"EdgeSGG's dual graph construction motivates the relation-node graph that HDGC combines with the heterogeneous graph.","marker":"[9]"},{"why":"IMP is the fully connected graph baseline whose wrong subject-object pair selection motivates the pair selection strategy.","marker":"[22]"},{"why":"Faster R-CNN provides the fixed object detector and proposals used in all experiments and comparisons.","marker":"[31]"},{"why":"BGNN is a main unbiased SGG baseline compared in Tables I and IV, providing adaptive message passing on bipartite graphs.","marker":"[13]"},{"why":"HL-Net is the prior state-of-the-art whose R@100 and mR@100 numbers TA-HDG claims to beat on all three Visual Genome subtasks.","marker":"[36]"},{"why":"RU-Net is the baseline whose graph-regularized subject-object pair filtering is compared against the proposed distance-confidence-existence strategy.","marker":"[25]"}],"fun_headline_variants":["Type-aware message passing lifts both rare and common scene relations","Splitting relation types boosts rare and common scene graph recall","Modeling relation-to-relation context lifts rare and common scene recall","Dual-graph message passing improves rare and common scene relations","Head and tail scene relations both improve with type-aware message passing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that every relation can be meaningfully divided into just two categories, interactive and non-interactive, and that this division is what rebalances the long tail; the paper does not list which of the 50 Visual Genome or 31 Open Images predicates fall into each category, so the split itself cannot be checked without going back to the cited Motifs super-types.","fun_headline_variants_meta":{"raw":{"variants":["Type-aware message passing lifts both rare and common scene relations","Splitting relation types boosts rare and common scene graph recall","Modeling relation-to-relation context lifts rare and common scene recall","Dual-graph message passing improves rare and common scene relations","Head and tail scene relations both improve with type-aware message passing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000685,"raw_usage":{"total_tokens":3106,"prompt_tokens":943,"completion_tokens":2163,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2091}},"tokens_in":559,"tokens_out":2163,"duration_ms":16737,"temperature":1.0,"reasoning_tokens":2091,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:36:38.916545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same TA-HDG pipeline with the relation-type assignment replaced by a random partition of the 50 Visual Genome predicates into two groups, preserving all other settings. If the mean-recall gains on SGCls remain close to the reported values, the type-aware split is not the operative mechanism; if they collapse, the split is load-bearing.","supporting_citations":[{"cited_title":"Neural motifs: Scene graph parsing with global context,","cited_arxiv_id":null,"evidence_quote":"Supplies the relation super-types, referenced as the basis for categorizing relations into interactive and non-interactive types."},{"cited_title":"Unbiased heterogeneous scene graph generation with relation-aware message passing neural network,","cited_arxiv_id":null,"evidence_quote":"HetSGG is the heterogeneous graph approach that HDGC extends, and it is the main previous baseline on mean recall."},{"cited_title":"Semantic Scene Graph Generation Based on an Edge Dual Scene Graph and Message Passing Neural Network","cited_arxiv_id":"2311.01192","evidence_quote":"EdgeSGG's dual graph construction motivates the relation-node graph that HDGC combines with the heterogeneous graph."},{"cited_title":"Hl-net: Heterophily learning network for scene graph generation,","cited_arxiv_id":null,"evidence_quote":"HL-Net is the prior state-of-the-art whose R@100 and mR@100 numbers TA-HDG claims to beat on all three Visual Genome subtasks."},{"cited_title":"Ru-net: Regularized unrolling network for scene graph generation,","cited_arxiv_id":null,"evidence_quote":"RU-Net is the baseline whose graph-regularized subject-object pair filtering is compared against the proposed distance-confidence-existence strategy."}],"review_version":1}