FDSG: Forecasting Dynamic Scene Graphs

Bodo Rosenhahn; Hao Cheng; Michael Ying Yang; Yi Yang; Yuren Cong

REVIEW 3 major objections 4 minor 50 references

FDSG: Forecasting Dynamic Scene Graphs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FDSG forecasts complete future scene graphs, including entity labels, boxes, and relationships, on previously unobserved frames, and reports gains over prior methods on Action Genome for generation, anticipation, and forecasting.

desk verdict Useful new task and solid gains on Action Genome, but entity forecasts rest on an unchecked split of NeuralSDE outputs, so treat the scene-graph coherence claims with caution. read the letter →

arxiv 2506.01487 v2 pith:FEBZ6IWT submitted 2025-06-02 cs.CV

Yi Yang , Yuren Cong , Hao Cheng , Bodo Rosenhahn , Michael Ying Yang This is my paper · ORCID

classification cs.CV

keywords dynamicscenegraphsgraphforecastinganticipationneuralstochasticdifferentialequationsvideounderstandingActionGenomeDINOquerydecompositiontemporalaggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

A scene graph is a structured snapshot of a video frame: nodes for objects, edges for their relationships. This paper argues that such a graph can be forecast, not merely reconstructed, for future frames, including entity labels, bounding boxes, and predicates together. To make the case, it introduces Scene Graph Forecasting (SGF), a task that scores all three outputs for unobserved frames, and FDSG, a one-stage model that evolves triplet representations in continuous time with a neural stochastic differential equation and then fuses forecasted with observed information. On the Action Genome dataset, the paper reports consistent gains over prior methods on dynamic scene graph generation, scene graph anticipation, and the new forecasting task. The point that would matter if true is that extrapolation is not a separate trick but an organizing objective that improves a video model's understanding of both the future and the observed present.

What carries the argument

The load-bearing object is a 512-dimensional triplet representation formed by concatenating DINO's decomposed content representations of the subject and object, with the predicate implicitly encoded. A Triplet Dynamics Model treats this vector as the initial value of a NeuralSDE, an integral equation whose neural-network drift and diffusion terms are driven by a Wiener process, and integrates it forward to forecast the triplet at a future time. The paper then reverses the concatenation to separate subject and object content, uses an identity-mapping Location Dynamics Model to carry the last observed query location forward, and feeds the forecast through the same classification heads used for observed frames. A Temporal Aggregation Module first cross-attends to observed reference triplets and then to forecasted triplets, on the grounds that observed information has lower uncertainty.

What would settle it

Swap the subject and object halves of the triplet vector at the forecast module's split point during training on Action Genome while keeping everything else fixed. If entity-label and bounding-box accuracy on SGF does not degrade, the split-recovery assumption is not load-bearing; if it degrades, the model depends on preserving the concatenation order.

Watch

Extended reading notes

Core claim

The paper introduces Scene Graph Forecasting (SGF): given observed frames up to time T, the model predicts complete scene graphs for future frames, including entity labels, bounding boxes, and pairwise relationships, rather than only predicate labels. It argues that this is achievable with a one-stage end-to-end model that represents each triplet as a concatenation of subject and object content vectors, evolves this 512-dimensional representation forward in continuous time with a NeuralSDE, and then recovers the two entity representations by splitting the forecast vector in half. A location dynamics model based on identity mapping carries the last observed query location forward, and a temporal aggregation module first attends to observed reference triplets and then to forecasted triplets. The reported experiments on Action Genome show consistent improvements over the compared methods on DSGG, SGA, and the newly proposed SGF task.

Load-bearing premise

The forecast step assumes that a dynamics network fed a subject-object vector glued together in a fixed order returns a vector whose first and second halves can be split apart to recover each entity's representation; because the network's drift and diffusion are unconstrained, nothing in the learned evolution guarantees that the concatenation order survives integration.

Editorial extensions

If this is right

If the central claim holds, SGF becomes a benchmarkable task: future frames can be scored for simultaneous correctness of entity labels, bounding box quality at IoU at least 0.5, and predicates, instead of predicate-only anticipation.
A model trained with a forecasting loss should keep or improve its observed-frame scene graph accuracy, so extrapolation can be used as an auxiliary objective rather than only a downstream capability.
One-stage detection-based scene graph models can forecast new entities rather than re-using entities from the last observed frame, without explicit object tracking.
The order of temporal aggregation matters: observed reference triplets should be integrated before higher-uncertainty forecasted triplets to maximize DSGG, SGA, and SGF accuracy.
Forecasting entity boxes through a reference box plus a content-predicted residual can support joint label, location, and relationship prediction from a single triplet representation.
A model trained with a forecasting loss should keep or improve its observed-frame scene graph accuracy, so extrapolation can be used as an auxiliary objective rather than only a downstream capability.
Because FDSG forecasts boxes through a reference-box-plus-residual mechanism, its box forecasts are anchored to the last observed location; extending the location model to full dynamics rather than identity mapping may improve long-horizon motion of fast-moving objects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same split-and-forecast design could be stress-tested on longer horizons and on datasets with faster motion; the paper's object-detection AP already trails an oracle as the forecasting window grows, suggesting error accumulation is the next bottleneck.
SGF could serve as a structured intermediate target for video world models: instead of predicting pixels, a model predicts a graph whose nodes and edges can be checked directly, which may make learned dynamics more interpretable.
The concatenation-split assumption could be probed by role-swap experiments or by forecasting subject and object with separate dynamics models; if those match FDSG, the compact triplet representation is not the source of its gains.
Because FDSG forecasts boxes through a reference-box-plus-residual mechanism, its box forecasts are anchored to the last observed location; extending the location model to full dynamics rather than identity mapping may improve long-horizon motion of fast-moving objects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, read this one for the task definition, not for the architectural rigor. FDSG introduces Scene Graph Forecasting—forecasting entity labels, boxes, and predicates for unobserved frames—and that is a genuinely missing capability. The paper is upfront that prior work predicts predicates with static entities (SceneSayer) or boxes without relations (object forecasting), and the SGF formulation cleanly fills that gap. On Action Genome, the reported gains over OED on SGDET and over SceneSayer on AGS/PGAGS/GAGS are consistent and often large, and the ablation table is informative. The qualitative sections are honest about failure cases, which I appreciate.

The soft spots are real. The main one, which the reader and I agree on, is Eq. (6): the TDM forecasts a concatenated 512-d triplet and then splits it in half to recover subject and object content. The NeuralSDE drift and diffusion are unconstrained MLPs. Nothing in the architecture forces the two halves to stay separable, and nothing forces the same entity to get a consistent forecast across the different triplets it appears in. The paper reports no consistency loss, no entity-level state, no merging step. So the claim that FDSG forecasts coherent future scene graphs is, as written, under-supported; it is forecasting a bag of locally plausible triplets that may not agree on entity labels or boxes. This is fixable—a shared entity query or an agreement loss would help—but it needs to be addressed, or at least honestly discussed. The ablation does not isolate this, because the full model is compared only against variants that remove entire modules.

A second, minor issue: the SGCLS prose says the model outperforms baselines, but in Table 3 FDSG ties DSG-DETR on R@50 No Constraint (72.4) and is best on other settings. That is a small overstatement, not a fatal one. Third, the SGF baselines are adapted from SceneSayer with two MLPs and look weak (single digits). That is partly because the task is new, and the paper says so. Still, the adapted baselines are the authors' own construction, so the SGF numbers should be read as an upper bound placeholder until others implement the task. Fourth, no error bars, no code, and only one dataset. For a task-definition paper, that is a significant lack—the field cannot anchor to SGF without a released benchmark.

Overall: the central idea is plausible and the experiments support it directionally. I would send this to review, but ask for code release, error bars, and a direct response to the Eq. (6) consistency issue. It is a solid workshop-to-conference paper that needs one more revision.

Referee Report

3 major / 4 minor

Summary. This paper proposes FDSG, a one-stage end-to-end model for dynamic scene graph generation that also introduces Scene Graph Forecasting (SGF), a new task in which the model predicts complete future scene graphs—entity labels, bounding boxes, and predicates—for unobserved frames. The architecture builds on DINO for entity query decomposition, uses a NeuralSDE-based Triplet Dynamics Model to evolve concatenated subject-object representations, and adds a Temporal Aggregation Module that fuses observed and forecasted information via cross-attention. Experiments on Action Genome report improvements over prior DSGG and SGA methods and large gains over adapted SceneSayer baselines on the proposed SGF task, together with ablations and qualitative visualizations.

Significance. If the results hold, the paper makes a useful contribution by moving dynamic scene graph generation from interpolation to extrapolation, proposing a concrete evaluation protocol for full future scene graphs, and showing evidence on Action Genome that forecasting objectives can also improve observed-frame DSGG. The paper is rich in experimental detail, including ablations of each module and of temporal aggregation strategies. The main risk is that entity-level forecasts are obtained by splitting an unconstrained triplet-level dynamics output, with no explicit mechanism ensuring that the same entity receives consistent forecasts across the different triplets in which it appears; this directly affects the central claim of forecasting coherent scene graphs.

major comments (3)

[§3.4.1, Eqs. (6), (12)] The entity-level forecasts rest on an unconstrained split of the NeuralSDE output. Because the drift μ_θc and diffusion ν_φc act on the full 512-d concatenated triplet, the first and second 256-d halves of z_{T0+ΔT} are not structurally guaranteed to correspond to the subject and object content, and each triplet is integrated separately. The same entity can therefore receive different content vectors—and hence different labels via Eq. (8) and different boxes via Eq. (13)—depending on the object it is paired with in a given forecasted frame. No consistency loss, entity-level state, or merging step is reported, so the claim of forecasting complete, globally coherent scene graphs is not yet supported. Please add an architectural constraint or an explicit entity-level forecast, or provide an empirical analysis of cross-triplet consistency (e.g., measure label/box disagreement for the same entity across its triplets) and its effect on the SGF metrics.
[§5.2, Tables 2–8] All quantitative results are reported for a single run, with no error bars or significance tests, and several headline comparisons rest on very small margins. For example, Table 4 shows FDSG at 72.9 versus OED at 73.0 for PredCLS R@10 With Constraint, and Table 3 shows FDSG at 59.0 versus DSG-DETR at 59.2 for SGCLS R@10 No Constraint and 67.2 versus 69.1 for R@20 No Constraint. The text in §5.2.1 says the model generally "outperforms other baselines," but the table shows DSG-DETR is not consistently worse. Please add standard deviations over multiple seeds and, for close comparisons, a paired significance test or an explicit statement that the differences are not statistically significant; also qualify the SGCLS claim.
[§4.3, Table 8] The SGF benchmark is introduced in this paper, and its only baselines in Table 8 are the authors' own adaptations of SceneSayer, created by attaching MLP heads to SceneSayer's predicate-only representations. This makes the large reported gains (e.g., R@10 With Constraint at F=0.5: 10.6 vs. 6.4) difficult to interpret, since the comparison conflates the proposed triplet-level representation with the new task itself. Please report the training protocol of the adapted baselines (losses, epochs, observation windows) and add at least one non-SceneSayer baseline, such as a static-copy detector or an OED-based forecasting variant, so that the SGF evaluation does not depend on a single baseline family.

minor comments (4)

[§3.4.1, Eq. (11)] The integrals in Eq. (11) use z_{T0} inside the integrands; as written this is not a standard Itô SDE. Please write μ_θc(z_t)dt + ν_φc(z_t)dW_t, or explicitly define the shorthand used.
[§5.2.1] The sentence "we have to exclude OED in Table 3" is confusing because another DETR-based method (DSG-DETR) is included in the same table; please clarify why OED cannot be compared and whether the evaluation code is shared across all baselines.
[§5.2.3] There is a typo in the sentence beginning "Notebly, in SceneSayer z represents only the predicate"; it should read "Notably."
[Table 1] The checkmark conventions in Table 1 are hard to parse; a legend defining what each symbol means for every column would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FDSG's central forecasting claim is trained and evaluated against external Action Genome baselines, and its prediction equations are architectural choices rather than reductions to the paper's own inputs.

full rationale

The paper's derivation chain is not circular. The Forecast Module uses a NeuralSDE to evolve the concatenated triplet representation, and Eqs. (6) and (12) split the forecast vector to recover entity content representations; this is a model-design assumption, not a tautology, because the NeuralSDE output is produced by learned drift/diffusion networks and is trained against future-frame ground truth, so the forecast is not equal to its input by construction. Even if the split is not structurally guaranteed to preserve entity coherence, that is an architectural or correctness concern, not a circularity. The SGF task is newly proposed by the authors, but it is evaluated with standard R@K/mR@K metrics and IoU-thresholded bounding-box matching on the external Action Genome dataset, and no constants are fitted to the reported test numbers. Baselines include independently published methods such as OED and SceneSayer, and the adapted SceneSayer+ variants are described with explicit equations (Eqs. (20)-(21)) rather than being silently set to favor FDSG. The paper does cite author-owned prior work, notably STTran [6] and RelTR [27], but these citations support generic design choices (triplet concatenation, match-and-assign evaluation) that are also grounded in external references such as EGTR [28] and SceneSayer [15]; they are not load-bearing for the core forecasting claim, and no uniqueness theorem or fitted parameter is imported from those citations. The central claim — that forecasting future scene graphs with entity labels, boxes, and relationships outperforms existing DSGG, SGA, and adapted SGF baselines — stands on independent training, external benchmarks, and reported comparisons, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on no new physical entities. The key assumptions are about query identity across time and the decomposability of forecasted triplet vectors. Free parameters are standard model capacities and loss weights chosen from prior practice.

free parameters (4)

Number of entity queries N_q = 100
Set to 100 following DETR-style detection; determines the maximum number of entities and triplets the model can output.
TopK reference triplet counts = {80n, 50n, 30n} with n=2
Selected following OED [11]; controls how many reference triplets are aggregated at each decoder level.
Matching loss weights alpha_l, beta_l = alpha_l = 1.0, beta_l = 5.0
Set for Hungarian matching; standard weighting, not derived.
Training epochs and learning rates = 5 epochs / 3 epochs, LR 0.0001
Chosen from prior training practices; not derived from first principles.

assumptions (4)

domain assumption Fixed DINO queries preserve entity identity across frames
The forecast evolves query representations from the last observed frame into the future; if query identity is not stable, the triplet dynamics have no consistent subject/object correspondence. Invoked in Sec. 3.4 and Eq. (12).
ad hoc to paper NeuralSDE output can be split into subject and object content vectors
Eq. (6) recovers entity content by splitting the forecasted triplet in half, but the neural drift and diffusion are not constrained to keep the two halves independent.
domain assumption Action Genome ground truth is complete and evaluation protocol matches prior work
All reported comparisons depend on the standard Action Genome splits and evaluation code used by OED and SceneSayer; no new annotation is provided.
standard math NeuralSDE numerical solver (reversible Heun) is accurate enough for the learned dynamics
Borrowed from SceneSayer [15]; numerical error is assumed small.

invented entities (1)

Scene Graph Forecasting (SGF) task independent evidence
purpose: Benchmark for predicting full future scene graphs, including entity labels, boxes, and predicates, from observed video frames.
The task is defined by the authors and evaluated on public Action Genome annotations. It is a new evaluation protocol rather than a physical entity; future work can independently train on the same data to verify the benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FDSG: Forecasting Dynamic Scene Graphs." pith.science (2026). https://pith.science/paper/FEBZ6IWT

@misc{pith2026250601487,
  author       = {Pith},
  title        = {Pith review of: FDSG: Forecasting Dynamic Scene Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FEBZ6IWT}},
  note         = {Machine review of arXiv:2506.01487}
}

read the original abstract

Dynamic scene graph generation extends scene graph generation from images to videos by modeling entity relationships and their temporal evolution. However, existing methods either generate scene graphs from observed frames without explicitly modeling temporal dynamics, or predict only relationships while assuming static entity labels and locations. These limitations hinder effective extrapolation of both entity and relationship dynamics, restricting video scene understanding. We propose Forecasting Dynamic Scene Graphs (FDSG), a novel framework that predicts future entity labels, bounding boxes, and relationships, for unobserved frames, while also generating scene graphs for observed frames. Our scene graph forecast module leverages query decomposition and neural stochastic differential equations to model entity and relationship dynamics. A temporal aggregation module further refines predictions by integrating forecasted and observed information via cross-attention. To benchmark FDSG, we introduce Scene Graph Forecasting, a new task for full future scene graph prediction. Experiments on Action Genome show that FDSG outperforms state-of-the-art methods on dynamic scene graph generation, scene graph anticipation, and scene graph forecasting. Codes will be released upon publication.

Figures

Figures reproduced from arXiv: 2506.01487 by the authors.

**Figure 2.** Our FDSG Framework carries out Dynamic Scene Graph Generation ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Illustration of the proposed SGF task. 3.2 Overview The overall workflow of our proposed method is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Visualization of bounding boxes and generated scene graphs from [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: Visualization of sampling points of the Deformable Attention in the last Cross-Attention layer of FDSG’s entity [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 6.** Figure 6: Visualization of Dynamic Scene Graph Generation ( [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Visualization of Dynamic Scene Graph Generation ( [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Visualization of Dynamic Scene Graph Generation ( [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 47 canonical work pages

[1]

Image retrieval using scene graphs,

J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3668–3678. 1

work page 2015
[2]

Human action recognition and prediction: A survey,

Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022. 1

work page 2022
[3]

(2.5+ 1) d spatio-temporal scene graphs for video question answering,

A. Cherian, C. Hori, T. K. Marks, and J. Le Roux, “(2.5+ 1) d spatio-temporal scene graphs for video question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 1, 2022, pp. 444–453. 1

work page 2022
[4]

Hig: Hierarchical interlace- ment graph approach to scene graph generation in video under- standing,

T.-T. Nguyen, P . Nguyen, and K. Luu, “Hig: Hierarchical interlace- ment graph approach to scene graph generation in video under- standing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 384–18 394. 1

work page 2024
[5]

Prob- abilistic semantic retrieval for surveillance videos with activity graphs,

Y. Chen, J. Wang, Y. Bai, G. Casta ˜n´on, and V . Saligrama, “Prob- abilistic semantic retrieval for surveillance videos with activity graphs,” IEEE Transactions on Multimedia , vol. 21, no. 3, pp. 704– 716, 2018. 1

work page 2018
[6]

Spatial-temporal transformer for dynamic scene graph gener- ation,

Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Y. Yang, “Spatial-temporal transformer for dynamic scene graph gener- ation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16 372–16 382. 1, 2, 4, 6, 7, 8, 9

work page 2021
[7]

Detecting human-object rela- tionships in videos,

J. Ji, R. Desai, and J. C. Niebles, “Detecting human-object rela- tionships in videos,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 8106–8116. 1, 2

work page 2021
[8]

Target adaptive context aggregation for video scene graph generation,

Y. Teng, L. Wang, Z. Li, and G. Wu, “Target adaptive context aggregation for video scene graph generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13 688–13 697. 1, 2, 6, 7 13 Fig. 5: Visualization of sampling points of the Deformable Attention in the last Cross-Attention layer of FDSG’s entity decoder for th...

work page 2021

Show all 50 references

[9]

Cross-modality time- variant relation learning for generating dynamic scene graphs,

J. Wang, J. Huang, C. Zhang, and Z. Deng, “Cross-modality time- variant relation learning for generating dynamic scene graphs,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 8231–8238. 1, 2

2023
[10]

End- to-end video scene graph generation with temporal propagation transformer,

Y. Zhang, Y. Pan, T. Yao, R. Huang, T. Mei, and C.-W. Chen, “End- to-end video scene graph generation with temporal propagation transformer,” IEEE Transactions on Multimedia , vol. 26, pp. 1613– 1625, 2023. 1, 2

2023
[11]

Oed: Towards one-stage end-to-end dynamic scene graph generation,

G. Wang, Z. Li, Q. Chen, and Y. Liu, “Oed: Towards one-stage end-to-end dynamic scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 938–27 947. 1, 2, 4, 5, 6, 7

2024
[12]

Sora: A review on background, technology, limitations, and opportunities of large vision models,

Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao et al. , “Sora: A review on background, technology, limitations, and opportunities of large vision models,” arXiv preprint arXiv:2402.17177, 2024. 1

2024 arXiv
[13]

A survey on evaluation of large language models,

Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wanget al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1–45, 2024. 1

2024
[14]

Dynamic scene graph generation via anticipatory pre-training,

Y. Li, X. Yang, and C. Xu, “Dynamic scene graph generation via anticipatory pre-training,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 13 874–13 883. 1, 2, 6, 7

2022
[15]

Towards scene graph anticipation,

R. Peddi, S. Singh, P . Singla, V . Gogateet al., “Towards scene graph anticipation,” in European Conference on Computer Vision. Springer, 2024, pp. 159–175. 1, 2, 3, 4, 5, 6, 7, 8, 9

2024
[16]

Dino: Detr with improved denoising anchor boxes for end-to-end object detection,

H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.- Y. Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,” arXiv preprint arXiv:2203.03605, 2022. 2, 4

2022 arXiv
[17]

Efficient and accurate gradients for neural sdes,

P . Kidger, J. Foster, X. C. Li, and T. Lyons, “Efficient and accurate gradients for neural sdes,” Advances in Neural Information Process- ing Systems, vol. 34, pp. 18 747–18 761, 2021. 2, 4

2021
[18]

Action genome: Actions as compositions of spatio-temporal scene graphs,

J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10 236–10 247. 2, 6

2020
[19]

Scene graph generation by iterative message passing,

D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, “Scene graph generation by iterative message passing,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5410–5419. 2

2017
[20]

Neural motifs: Scene graph parsing with global context,

R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, “Neural motifs: Scene graph parsing with global context,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5831–

2018
[21]

Gps-net: Graph property sensing network for scene graph generation,

X. Lin, C. Ding, J. Zeng, and D. Tao, “Gps-net: Graph property sensing network for scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3746–3753. 2, 6, 7

2020
[22]

Unbiased scene graph generation from biased training,

K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, 15 Fig. 8: Visualization of Dynamic Scene Graph Generation ( DSGG) and Scene Gra...

2020
[23]

Ru-net: Regular- ized unrolling network for scene graph generation,

X. Lin, C. Ding, J. Zhang, Y. Zhan, and D. Tao, “Ru-net: Regular- ized unrolling network for scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022, pp. 19 457–19 466. 2

2022
[24]

Faster r-cnn: Towards real-time object detection with region proposal networks,

S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1137–1149, 2016. 2

2016
[25]

End-to-end object detection with transformers,

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision. Springer, 2020, pp. 213–229. 2, 5

2020
[26]

Sgtr: End-to-end scene graph genera- tion with transformer,

R. Li, S. Zhang, and X. He, “Sgtr: End-to-end scene graph genera- tion with transformer,” in proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 19 486–19 496. 2

2022
[27]

Reltr: Relation trans- former for scene graph generation,

Y. Cong, M. Y. Yang, and B. Rosenhahn, “Reltr: Relation trans- former for scene graph generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 9, pp. 11 169–11 183,
[28]

Egtr: Extracting graph from transformer for scene graph generation,

J. Im, J. Nam, N. Park, H. Lee, and S. Park, “Egtr: Extracting graph from transformer for scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24 229–24 238. 2, 4, 7

2024
[29]

Exploiting long-term dependencies for generating dynamic scene graphs,

S. Feng, H. Mostafa, M. Nassar, S. Majumdar, and S. Tripathi, “Exploiting long-term dependencies for generating dynamic scene graphs,” in Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, 2023, pp. 5130–5139. 2, 6, 7, 8, 9

2023
[30]

Multiple object forecasting: Predicting future object locations in diverse environments,

O. Styles, V . Sanchez, and T. Guha, “Multiple object forecasting: Predicting future object locations in diverse environments,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 690–699. 2

2020
[31]

Multiple object tracking and forecasting: Jointly predicting current and future object locations,

O. Kesa, O. Styles, and V . Sanchez, “Multiple object tracking and forecasting: Jointly predicting current and future object locations,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 560–569. 2

2022
[32]

A review of object tracking methods: From general field to autonomous vehicles,

J. Cao, H. Zhang, L. Jin, J. Lv, G. Hou, and C. Zhang, “A review of object tracking methods: From general field to autonomous vehicles,” Neurocomputing, 2024. 2

2024
[33]

Accurate optical flow via direct cost volume processing,

J. Xu, R. Ranftl, and V . Koltun, “Accurate optical flow via direct cost volume processing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1289–1297. 2

2017
[34]

Predicting the future: A jointly learnt model for action anticipation,

H. Gammulle, S. Denman, S. Sridharan, and C. Fookes, “Predicting the future: A jointly learnt model for action anticipation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5562–5571. 2

2019
[35]

Action anticipation by predicting future dynamic images,

C. Rodriguez, B. Fernando, and H. Li, “Action anticipation by predicting future dynamic images,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018. 2

2018
[36]

Future transformer for long-term action anticipation,

D. Gong, J. Lee, M. Kim, S. J. Ha, and M. Cho, “Future transformer for long-term action anticipation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3052–3061. 2

2022
[37]

Rethinking learning ap- proaches for long-term action anticipation,

M. Nawhal, A. A. Jyothi, and G. Mori, “Rethinking learning ap- proaches for long-term action anticipation,” in European Conference on Computer Vision. Springer, 2022, pp. 558–576. 2

2022
[38]

Adamsformer for spatial action localization in the future,

H.-g. Chi, K. Lee, N. Agarwal, Y. Xu, K. Ramani, and C. Choi, “Adamsformer for spatial action localization in the future,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 885–17 895. 2, 5

2023
[39]

Neural ordinary differential equations,

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” Advances in neural infor- mation processing systems, vol. 31, 2018. 2, 4

2018
[40]

Transvod: End-to-end video object detection with spatial- temporal transformers,

Q. Zhou, X. Li, L. He, Y. Yang, G. Cheng, Y. Tong, L. Ma, and D. Tao, “Transvod: End-to-end video object detection with spatial- temporal transformers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7853–7869, 2022. 4

2022
[41]

Focal loss for dense object detection,

T.-Y. Ross and G. Doll ´ar, “Focal loss for dense object detection,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2980–2988. 5

2017
[42]

Graphical contrastive losses for scene graph parsing,

J. Zhang, K. J. Shih, A. Elgammal, A. Tao, and B. Catanzaro, “Graphical contrastive losses for scene graph parsing,” in Pro- 16 ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 535–11 543. 6, 7

2019
[43]

Learning to com- pose dynamic tree structures for visual contexts,

K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to com- pose dynamic tree structures for visual contexts,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 6619–6628. 6, 7

2019
[44]

Unbiased scene graph generation in videos,

S. Nag, K. Min, S. Tripathi, and A. K. Roy-Chowdhury, “Unbiased scene graph generation in videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 803–22 813. 6, 7

2023
[45]

A comprehensive survey of scene graphs: Generation and applica- tion,

X. Chang, P . Ren, P . Xu, Z. Li, X. Chen, and A. Hauptmann, “A comprehensive survey of scene graphs: Generation and applica- tion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 1–26, 2021. 6

2021
[46]

Scene graph generation: A comprehensive survey,

H. Li, G. Zhu, L. Zhang, Y. Jiang, Y. Dang, H. Hou, P . Shen, X. Zhao, S. A. A. Shah, and M. Bennamoun, “Scene graph generation: A comprehensive survey,” Neurocomputing, vol. 566, p. 127052, 2024. 6

2024
[47]

Microsoft coco: Common objects in context,

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan, P . Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision, 2014, pp. 740–

2014
[48]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. 6

2016
[49]

Decoupled weight decay regulariza- tion,

I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” arXiv preprint arXiv:1711.05101, 2017. 6

2017 arXiv
[50]

Deformable detr: Deformable transformers for end-to-end object detection,

X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” in International Conference on Learning Representations, 2021. 12

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Image retrieval using scene graphs,

J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3668–3678. 1

work page 2015

[2] [2]

Human action recognition and prediction: A survey,

Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022. 1

work page 2022

[3] [3]

(2.5+ 1) d spatio-temporal scene graphs for video question answering,

A. Cherian, C. Hori, T. K. Marks, and J. Le Roux, “(2.5+ 1) d spatio-temporal scene graphs for video question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 1, 2022, pp. 444–453. 1

work page 2022

[4] [4]

Hig: Hierarchical interlace- ment graph approach to scene graph generation in video under- standing,

T.-T. Nguyen, P . Nguyen, and K. Luu, “Hig: Hierarchical interlace- ment graph approach to scene graph generation in video under- standing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 384–18 394. 1

work page 2024

[5] [5]

Prob- abilistic semantic retrieval for surveillance videos with activity graphs,

Y. Chen, J. Wang, Y. Bai, G. Casta ˜n´on, and V . Saligrama, “Prob- abilistic semantic retrieval for surveillance videos with activity graphs,” IEEE Transactions on Multimedia , vol. 21, no. 3, pp. 704– 716, 2018. 1

work page 2018

[6] [6]

Spatial-temporal transformer for dynamic scene graph gener- ation,

Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Y. Yang, “Spatial-temporal transformer for dynamic scene graph gener- ation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16 372–16 382. 1, 2, 4, 6, 7, 8, 9

work page 2021

[7] [7]

Detecting human-object rela- tionships in videos,

J. Ji, R. Desai, and J. C. Niebles, “Detecting human-object rela- tionships in videos,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 8106–8116. 1, 2

work page 2021

[8] [8]

Target adaptive context aggregation for video scene graph generation,

Y. Teng, L. Wang, Z. Li, and G. Wu, “Target adaptive context aggregation for video scene graph generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13 688–13 697. 1, 2, 6, 7 13 Fig. 5: Visualization of sampling points of the Deformable Attention in the last Cross-Attention layer of FDSG’s entity decoder for th...

work page 2021