{"id":"e48b1e20-8d75-4c2a-9c40-5da82023364e","arxiv_id":"2412.02808","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An end-to-end transformer couples detection with a temporal matching penalty and feedback queries, boosting temporal consistency of scene-graph predictions on Action Genome, OpenPVSG, and MEVA.","lead":"This paper presents TCDSG, an end-to-end model that detects objects and their interactions in video while keeping the same identity across frames, producing continuous action tracklets. It reports large gains on a temporal recall metric over a single-frame baseline and releases new identity annotations for part of the MEVA surveillance dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline tR@50 gain on Action Genome rests on IoU-pseudo-labeled object IDs shared by training and evaluation; identity switches or re-entries can inflate apparent temporal consistency.","rationale":"The paper's central claim is that TCDSG produces temporally consistent action tracklets end-to-end, evidenced most strongly by tR@50 improving from 18.6 to 39.1 on Action Genome. For that evidence to support the claim, the tR@K ground truth must reflect true object identities. The authors generate those identities by IoU overlap pseudo-labeling (Sec. 4.1), and the same pseudo-labels steer the sequence-level matching objective during training. This creates a circularity risk: the metric rewards stability of query identity relative to labels that are themselves generated by an IoU chaining heuristic, so the model may be rewarded for matching the heuristic rather than for tracking physical objects across occlusions, re-entries, or interactions between same-class objects. The paper's own limitation statement acknowledges the risk, and the absence of independent labels, error bars, or released code makes it impossible to quantify. Because this concern directly targets the headline evidence, it is load-bearing. It does not change the reader's CONDITIONAL verdict: the method may still be sound, but the central temporal-consistency claim on Action Genome is not yet established. I agree with the reader's weakest_assumption. Secondary issues, such as the vague description of Eq. 1 and the abstract's '60%' claim conflicting with the '+110%' in Sec. 5.1, are real but less consequential than the pseudo-labeling issue.","tokens_in":13010,"tokens_out":3495,"duration_ms":41388,"concrete_test":"Recompute the Action Genome tR@50 numbers on a held-out subset using independently assigned persistent IDs (e.g., a high-accuracy MOT tracker or manual instance labeling) instead of IoU pseudo-labels, for both TCDSG and DDS. If TCDSG's roughly 20-point advantage over DDS shrinks to a few points or to noise, the temporal-consistency claim is largely an artifact of shared pseudo-labels; if the advantage persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is the Action Genome pseudo-labeling in Sec. 4.1. AG provides no persistent object IDs, so the authors assign cross-frame identities by bounding-box IoU overlap. These pseudo-tracklets serve double duty: they are the training targets for the SLBM lock (Sec. 3.4, Eq. 3) and the ground truth for the tR@K metric (Sec. 3.6). Any IoU link that merges two same-class objects or swaps identities after occlusion makes \"temporal consistency\" mean consistency with a pseudo-ID rather than with a physical object. The headline tR@50 gain (18.6 to 39.1) could therefore partly reflect the model learning the same IoU-chaining heuristic that generated the labels, not truly identity-stable tracking. The paper acknowledges this risk in Sec. 4.1, citing manual verification, but no error bars, pseudo-labeling code, or independent labels are provided. OpenPVSG has true IDs, but its absolute tR@50 is only 14.2, and MEVA has no external baseline, so Action Genome is the only dataset carrying the strong claim, and it is the one with the least reliable identity signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TCDSG, an end-to-end transformer-based framework for dynamic scene graph generation and action tracklet construction. Two core mechanisms are introduced: a Sequence-Level Bipartite Matching (SLBM) objective that caches ground-truth triplet-to-query assignments and adds a penalty for identity switches, and Temporally Conditioned Decoder Queries (TCQ) that feed LSTM-summarized decoder outputs from the previous frame into the current frame's query embeddings. The authors evaluate on Action Genome, OpenPVSG, and a newly re-annotated MEVA subset, reporting large temporal Recall@K gains, notably tR@50 improving from 18.6% (DDS) to 39.1% on Action Genome. The paper also proposes new persistent object-ID annotations for a subset of MEVA.","tokens_in":13232,"tokens_out":3911,"duration_ms":42887,"significance":"If the results are reliable, the paper addresses a real limitation of frame-wise dynamic scene graph methods: tracklet fragmentation caused by query reassignment between frames. The SLBM idea is simple and plausible, and the ablations in Table 4 show a consistent trend that each proposed component contributes to temporal consistency. The paper is also honest in disclosing the pseudo-labeling strategy on Action Genome and the fact that MEVA results are on a newly curated subset with non-standard metrics. However, the headline evidence is weakened by the shared pseudo-label approximation between training and evaluation on Action Genome, the absence of the full TCDSG model on OpenPVSG, and the lack of any external baseline on MEVA. These issues are load-bearing because the paper claims state-of-the-art temporal consistency on three datasets.","major_comments":[{"comment":"The Action Genome ground-truth tracklet identities are pseudo-labeled by bounding-box IoU overlap (Sec. 4.1), and these same pseudo-IDs are used both as training targets for the SLBM lock (Eq. 3) and as ground truth for the tR@K metric (Sec. 3.6). This shared approximation can inflate temporal consistency: if the IoU chaining merges or swaps distinct same-class objects, a model that learns to reproduce the chaining heuristic will score highly on tR@K without achieving true identity-stable tracking. The manual verification mentioned in Sec. 4.1 is not quantified, and no independent labels or error bars are provided. I request a quantitative analysis of pseudo-label noise (e.g., agreement statistics on a manually labeled subset) or an evaluation on a dataset with true persistent IDs, such as the full TCDSG model on OpenPVSG.","section":"Sec. 4.1, Sec. 3.4, Sec. 3.6"},{"comment":"The OpenPVSG results report only TCDSG*, which Table 1 defines as the variant with Temporal Hungarian Matching removed. The full TCDSG model is never evaluated on OpenPVSG, so the paper does not actually demonstrate the proposed method's temporal consistency gains on this dataset. The authors should report the full model's R@K and tR@K on OpenPVSG and, if possible, include external baselines that report the same tR@K metric.","section":"Table 2, Sec. 4.2"},{"comment":"The MEVA evaluation contains no external baseline; Table 3 compares only subsampling factors of the authors' own model on their re-annotated subset. As the paper acknowledges, prior MEVA work uses different metrics, but this means the claim of strong temporal consistency on MEVA is supported only by an internal analysis. A comparison with at least one existing method under a common metric, or a clear statement that the MEVA contribution is a dataset/annotation resource rather than a performance claim, is needed.","section":"Sec. 5.1, Table 3"},{"comment":"The abstract claims 'over 60% improvement in temporal recall@k on the Action Genome, OpenPVSG, and MEVA datasets,' whereas Sec. 5.1 reports '+110% rel.' for the Action Genome tR@50 improvement (18.6 to 39.1) and no comparable baseline improvement is shown for OpenPVSG or MEVA. These statements are inconsistent and the abstract overstates what the tables support. The authors should reconcile the numbers and restrict the improvement claim to the comparisons actually reported.","section":"Abstract and Sec. 5.1"}],"minor_comments":[{"comment":"The description of the cross-attention operation f(a, b) is ambiguous: 'using a as input queries and b as query and value' is unclear about the roles of Q, K, and V, and b is described as an LSTM state rather than a sequence of keys/values. Please specify the exact tensor shapes and attention formulation.","section":"Eq. (1)"},{"comment":"The tR@K metric is described only verbally as requiring 'sufficient temporal Intersection over Union with ground truth.' The definition should be formalized: how is temporal IoU computed, what threshold is used, and how are tracklet boundaries aligned?","section":"Sec. 3.6"},{"comment":"There are frequent typos and spacing errors, including 'MEV A' instead of 'MEVA', 'reducing improving prediction stability' in Sec. 1, and inconsistent capitalization in 'The Reference point loss'. A careful proofread is needed.","section":"Throughout"},{"comment":"The TCDSG* rows are defined only in the caption of Table 1 and in the ablation text; it would help to state explicitly in each table caption that TCDSG* denotes removal of the temporal matching component.","section":"Tables 1 and 4"},{"comment":"No error bars, standard deviations, or multiple-seed results are reported anywhere. Given the reliance on pseudo-labels and the small MEVA test subset, at least a variance estimate over seeds or annotation subsets would materially strengthen the temporal consistency claims.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is the Action Genome pseudo-label circularity: the same IoU-chained identities are used for training supervision and evaluation, so the headline tR@K gain could partly reflect the model learning the label-generation heuristic. The missing full-model result on OpenPVSG and the absence of any MEVA baseline make the three-dataset claim fragile. I would suggest the editor ask for either an independent identity evaluation or a substantial toning-down of the headline claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: TCDSG is a solid, genuinely incremental contribution. The sequence-level bipartite matching that locks query-triplet identities across frames (SLBM) and the LSTM-conditioned decoder queries (TCQ) are new combinations in the DETR-based dynamic scene graph line, and the ablations on Action Genome show consistent tR@K gains from each added component. The paper is worth engaging.\n\nWhat it does well: the SLBM idea is clean—a small penalty term in Hungarian matching that caches triplet-to-query assignments and penalizes switches. It directly addresses tracklet fragmentation without post-processing. The TCQ feedback loop via LSTM is a sensible upgrade over simply feeding previous outputs. The OpenPVSG results are compared against external baselines (IPS+T, VPS, MACL), which is meaningful. The MEVA re-annotation with persistent IDs is a useful resource for the community.\n\nSoft spots: the headline AG numbers rest on IoU pseudo-labels because Action Genome has no persistent object IDs. The paper discloses this and cites manual verification, but the same pseudo-labels are used for training and evaluation, so tR@K can reward consistency toward wrong identities when same-class objects re-enter or occlude. This is the main caveat, and it is a real one. Second, the abstract claims 'over 60% improvement' while Sec 5.1 reports +110% relative on the same number (18.6 to 39.1). That inconsistency is sloppy and should be fixed. Third, no error bars, and no code/data are provided. The MEVA evaluation has no external baseline and the full-frame-rate tR@50 is only 7.8; the subsampling analysis is interesting but does not change the fact that absolute numbers are low.\n\nNone of these are fatal. The central mechanism is plausible, the ablations support it, and the limitations are at least acknowledged. What the paper needs is a round of revision that adds error bars, releases code and pseudo-labeling details, aligns the abstract with the actual numbers, and ideally evaluates the impact of the pseudo-label assumption (e.g., on OpenPVSG where true IDs exist).\n\nFor a reader in video scene understanding, tracking, or surveillance analytics, this is worth a look. I would send it to peer review rather than desk-reject, with a request for the above revisions.","headline":"A genuinely new end-to-end temporal consistency mechanism for video scene graphs with honest ablations, but the headline AG numbers rest on IoU pseudo-labels and the abstract overclaims.","tokens_in":13820,"tokens_out":2160,"would_cite":true,"duration_ms":20871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TCDSG claims that a transformer video scene-graph detector trained with a temporal matching penalty and LSTM-conditioned queries yields identity-stable action tracklets end-to-end, lifting Action Genome tR@50 from 18.6% to 39.1%.","keywords":["dynamic scene graphs","action tracklets","temporal recall K","sequence-level bipartite matching","temporally conditioned queries","video scene understanding","LSTM feedback","persistent object IDs"],"falsifier":"Re-measure Action Genome tR@50 using independently verified tracklet identities, for example human annotations or a separate strong tracker, instead of the paper's IoU-overlap pseudo-labels; if the gap between TCDSG and the DDS baseline narrows sharply, part of the reported gain is an artifact of shared label noise. On the re-annotated MEVA subset, the analogous check is to recompute tR@K while comparing predicted query groups against the provided persistent IDs rather than against groups formed by the same pseudo-linking rule.","tokens_in":12780,"feed_emoji":"🎬","tokens_out":9101,"duration_ms":84417,"temperature":0.7,"pith_summary":"This paper tries to establish that action tracklets can be produced end-to-end by making the detector itself temporally consistent, instead of stitching per-frame scene graphs together afterward. The two mechanisms claimed to do the work are a sequence-level matching objective that locks each ground-truth interaction to a single decoder slot across frames, and decoder queries that are updated with a recurrent memory of previous frames' outputs. The headline evidence is temporal Recall@50 rising from 18.6% for a framewise baseline to 39.1% on Action Genome, with gains also reported on OpenPVSG and a newly re-annotated MEVA subset. If the claim is right, continuous video tasks such as surveillance and autonomous navigation could read stable subject-object-relation tracklets directly from one transformer pipeline.","feed_headline":"Temporal matching lifts tracklet recall from 18.6% to 39.1%","feed_subtitle":"Keeping one decoder query per interaction across frames lets action tracklets emerge directly from the detector","key_machinery":"The load-bearing mechanism is Sequence-Level Bipartite Matching (SLBM): a cache-and-penalize extension of Hungarian matching, the optimal one-to-one assignment between predicted slots and ground-truth objects. A hash map records which decoder query first won each ground-truth triplet $\\langle s, o, r\\rangle$; in every later frame, any cost-matrix entry that would assign that triplet to a different query receives a large penalty, so the matching is biased to keep identities fixed across time. The second mechanism is Temporally Conditioned Queries (TCQ): decoder queries are formed by cross-attending the learnable query embeddings to an LSTM state computed from the previous frame's decoder outputs, giving the decoder memory, context, and continuity. Together these make query index a stable proxy for object-triplet identity, which is exactly what lets tracklet construction be a simple grouping step.","core_discovery":"The central claim is that temporal coherence can be trained into a DETR-style scene-graph detector instead of being added as post-processing. TCDSG's Sequence-Level Bipartite Matching objective caches each ground-truth triplet the first time it appears, binds it to the decoder query it was matched to, and adds a large penalty $\\lambda_{\\mathrm{co}}$ to any later Hungarian assignment that would move the triplet to another query. Alongside it, Temporally Conditioned Queries update the decoder's input queries as $Q_t = f(Q_{\\mathrm{learn}}, \\mathrm{LSTM}(D_{t-1}))$, giving the decoder a learned memory of prior frames. At inference the predictions are grouped by query index, and because the training objective has suppressed query-to-triplet reassignment, those groups form action tracklets without IoU linking or re-identification. On Action Genome the authors report tR@50 improving from 18.6% for the DDS baseline to 39.1%, while frame-level R@50 stays competitive at 52.5%.","pith_inferences":["The paper leaves implicit that the first-match lock is permanent: if a triplet is assigned to the wrong query in an early frame, SLBM will keep that wrong assignment alive for the whole visible tracklet, and a confidence-gated commitment or delayed locking would be a testable remedy.","Because Action Genome's pseudo-IDs are shared by training and evaluation, the tR@K gain partly measures consistency toward the approximation; a metric that also requires the tracked identity to match an independently verified ID would separate consistency from correctness.","The matching-level lock is independent of the detector's class vocabulary, so the same assignment mechanism plausibly transfers to open-vocabulary or language-prompted video scene understanding, with queries conditioned on text embeddings."],"forward_implications":["Tracklet construction becomes a grouping step: predictions that share a query index and triplet label across consecutive frames are concatenated directly, with no IoU linking or re-identification post-processing.","Temporal consistency is bought at a measurable cost to single-frame accuracy: removing temporal locking raises R@50 from 52.5% to 58.4% on Action Genome while tR@50 drops from 39.1% to 27.1%.","Long, densely sampled videos expose transient query switches: on the re-annotated MEVA subset, keeping every 20th frame lifts tR@50 from 7.8% at full frame rate to 31.2%.","The re-annotated MEVA subset, with 141 training and 47 test videos across multiple cameras, provides a benchmark for long-horizon tracklet generation in real surveillance footage."],"supporting_citations":[{"why":"Supplies the DDS baseline that TCDSG extends and the 18.6% Action Genome tR@50 comparison point.","marker":"[19]"},{"why":"Supplies the DETR set-prediction and per-frame Hungarian matching formulation that SLBM modifies.","marker":"[3]"},{"why":"Supplies the Deformable DETR backbone and reference-point mechanism that the decoder branches build on.","marker":"[64]"},{"why":"Action Genome provides the frame-level scene-graph annotations that TCDSG pseudo-labels into tracklet identities.","marker":"[21]"},{"why":"Defines action tracklets and the OpenPVSG benchmark and supplies the post-processing-based comparison baselines.","marker":"[59]"},{"why":"MEVA is the large-scale multi-camera dataset the paper re-annotates with persistent object IDs for long-range evaluation.","marker":"[10]"}],"fun_headline_variants":["End-to-end scene graphs generate action tracklets directly","Sequence-level matching lifts temporal recall to 39.1%","No post-processing: tracklets from dynamic scene graphs","From 18.6% to 39.1%: temporal coherence in decoder queries","Adaptive query memory yields consistent action tracklets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the IoU-overlap pseudo-labels on Action Genome give correct ground-truth tracklet identities; if same-class objects re-enter or occlude each other, the pseudo-IDs merge or swap, and because the same labels drive both training and evaluation the reported tR@K gain can reward stable-but-wrong identities.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end scene graphs generate action tracklets directly","Sequence-level matching lifts temporal recall to 39.1%","No post-processing: tracklets from dynamic scene graphs","From 18.6% to 39.1%: temporal coherence in decoder queries","Adaptive query memory yields consistent action tracklets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3711,"prompt_tokens":959,"completion_tokens":2752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2667}},"tokens_in":575,"tokens_out":2752,"duration_ms":22200,"temperature":1.0,"reasoning_tokens":2667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:04:22.176031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-measure Action Genome tR@50 using independently verified tracklet identities, for example human annotations or a separate strong tracker, instead of the paper's IoU-overlap pseudo-labels; if the gap between TCDSG and the DDS baseline narrows sharply, part of the reported gain is an artifact of shared label noise. On the re-annotated MEVA subset, the analogous check is to recompute tR@K while comparing predicted query groups against the provided persistent IDs rather than against groups formed by the same pseudo-linking rule.","supporting_citations":[{"cited_title":"Dds: Decoupled dynamic scene-graph generation network","cited_arxiv_id":null,"evidence_quote":"Supplies the DDS baseline that TCDSG extends and the 18.6% Action Genome tR@50 comparison point."},{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the DETR set-prediction and per-frame Hungarian matching formulation that SLBM modifies."},{"cited_title":"Action genome: Actions as compositions of spatio- temporal scene graphs","cited_arxiv_id":null,"evidence_quote":"Action Genome provides the frame-level scene-graph annotations that TCDSG pseudo-labels into tracklet identities."},{"cited_title":"Panoptic video scene graph generation","cited_arxiv_id":null,"evidence_quote":"Defines action tracklets and the OpenPVSG benchmark and supplies the post-processing-based comparison baselines."},{"cited_title":"Meva: A large-scale multiview, multimodal video dataset for activity detection","cited_arxiv_id":null,"evidence_quote":"MEVA is the large-scale multi-camera dataset the paper re-annotates with persistent object IDs for long-range evaluation."}],"review_version":1}