{"id":"b00e7d31-59c5-4b69-bf13-1d1792243d51","arxiv_id":"2505.06663","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A unified query-based model that alternately refines object and relationship representations achieves new best mAP on VidVRD and VidOR open-vocabulary video relationship detection.","lead":"METOR is a video model that detects objects and their relationships at the same time, letting each task improve the other instead of running one after the other. It reports better results on two standard video relationship benchmarks, though the gains over the closest previous method are small.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The open-vocabulary gain is attributed to CLIP-based contextual refinement, but the ablations remove whole modules and the contextual losses are self-referential, so the gain could be extra capacity rather than the proposed mechanism.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the contextual-refinement module's open-vocabulary benefit is only shown by removing whole modules, so extra capacity is a plausible confound. My stress-test sharpens this by noting that the contextual losses in Eq. 16 are self-referential, meaning they cannot independently certify that Co/Cr encode category-presence information. This is the appropriate focus because the paper's novelty, and its claimed improvement over EOV-MMP, rests on this mechanism; if the gains come only from added parameters, the stated contribution is not established. The concern is not a rejection: the architecture is coherent, the ablations are directionally consistent, and a direct probe or capacity-matched baseline could settle it. Since the reader already conditioned the verdict on the same class of concern, no change in verdict is needed.","tokens_in":11579,"tokens_out":6325,"duration_ms":74347,"concrete_test":"Train a linear or logistic probe on top of the frozen Co (and Cr) embeddings from the trained METOR model to predict which base object (relationship) categories are present in a frame, then evaluate probe recall on held-out novel categories. As a capacity control, replace co/cr with learnable tokens that are concatenated after the frozen CLIP encoder rather than before it, keeping the same number of parameters, Mo/Mr, and all losses; if the control matches METOR's novel-split mAP and transferable probe accuracy, the improvement is attributable to added capacity, not to CLIP-based contextual refinement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that METOR improves open-vocabulary generalization through CLIP-based contextual refinement presupposes that the learnable tokens co and cr (Sec. 3.3, Eqs. 7-10) produce context embeddings Co and Cr that carry category-presence information for novel objects and relationships. The evidence offered is primarily Table 3, which removes the entire contextual-refinement module or its subparts. Because co, cr, Mo, and Mr are trained jointly with all classification losses, the observed improvement (e.g., novel-split mAP 16.43 vs. 13.49 for w/o CRE on VidVRD) could stem from added trainable parameters and extra attention capacity rather than from any semantically meaningful use of CLIP's visual features. Eq. 16's contextual losses do not resolve this: Lobj-ctx aligns Co with To, but To is itself computed from Co via Eq. 9, so the target is not independent of the predictor; the loss can be minimized without Co encoding novel-category presence. Similarly for Cr and Tr via Eq. 10. Thus the open-vocabulary mechanism, which is the core novelty beyond EOV-MMP, is not yet isolated from a capacity confound. Without an independent probe of Co/Cr or a capacity-matched ablation, the central attribution claim remains insecure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes METOR, a query-based unified framework for open-vocabulary video visual relationship detection (Open-VidVRD). The framework contains two main components: a CLIP-based contextual refinement encoding module that appends learnable object and relationship tokens to a frozen CLIP ViT visual encoder, uses the resulting context embeddings to refine object queries and text features, and a spatio-temporal iterative enhancement module that alternately refines subject, object, and relationship features. The model is trained end-to-end with object/relationship contrastive losses, a trajectory loss, and contextual losses. Experiments on VidVRD and VidOR under novel-split and all-split SGDet settings report improvements over the prior EOV-MMP method on most metrics, with ablations on VidVRD attributing gains to the two proposed modules.","tokens_in":11887,"tokens_out":4713,"duration_ms":51590,"significance":"If the claims are validated, the paper makes a useful contribution by moving Open-VidVRD away from cascaded pipelines and toward a unified query-based formulation, with a concrete mechanism for mutual enhancement of object and relationship representations. The authors provide code, an end-to-end design, and relatively large gains on the novel split of VidOR (relative mAP improvement of about 53% over EOV-MMP), which is the practically important setting for open-vocabulary generalization. However, the central attribution of these gains to the proposed contextual-refinement mechanism is not fully isolated: the ablations remove whole modules, the contextual losses do not provide an independent probe of the learned context embeddings, and all results are single-run. The comparison with the strongest baseline, EOV-MMP, is also a comparison with the authors' own prior work, which reduces the strength of the headline comparison.","major_comments":[{"comment":"","section":"Sec. 3.3, Eqs. (7)-(10) and Eq. (16)"},{"comment":"","section":"Table 1"},{"comment":"","section":"Tables 2-4"}],"minor_comments":[{"comment":"","section":"Table 1 and Related Work"},{"comment":"","section":"Fig. 1(c)"},{"comment":"","section":"Sec. 4.2"},{"comment":"","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is that the open-vocabulary contribution is not yet isolated from a capacity confound; this is fixable with additional experiments, so I do not recommend rejection. I also note that the strongest baseline in the VidOR comparison is the authors' own prior work (EOV-MMP), so an independent evaluation or a third-party reimplementation would materially strengthen the paper. The paper fits the scope of the journal, and the proposed framework is a reasonable extension of the authors' earlier EOV-MMP line, but the novelty over that prior work should be stated more crisply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on METOR (arXiv:2505.06663): it's a competent, incremental extension of the same group's EOV-MMP, and it does report real gains on the SGDet task for VidVRD and VidOR, mainly on mAP and mAPo. The architecture is new in the details: a query-based unified model that alternates between enhancing object and relationship representations, plus learnable context tokens appended to CLIP to refine both object queries and text features. The ablations are consistent and the writing is clear. If you work on Open-VidVRD, this is worth reading.\n\nWhat works well: the problem framing is honest—cascaded pipelines do suffer from error propagation, and the error analysis in Fig. 1 is a nice touch. The iterative enhancement module is simple and seems to help (Table 4 peaks at two iterations, and the gains are not monotonic, which is a good sanity signal). The module-level ablations in Table 2 show that each component contributes, and the qualitative T-SNE figures are supportive, though not conclusive. Code is released, which I count as a plus.\n\nThe soft spots are real and mostly in the attribution of the open-vocabulary gain. The paper's core novelty is the contextual refinement encoding module (Sec 3.3, Eq 7-10). The stress-test note is on target: the contextual losses in Eq 16 are self-referential. To is computed from Co via Eq 9, then Lobj-ctx aligns Co with To. That loss can be minimized without Co encoding any category-presence signal. Meanwhile, adding learnable tokens and mapping layers to a frozen CLIP encoder adds capacity, so the ablation \"w/o CRE\" (Table 3) removes the whole module and cannot separate a capacity effect from the semantic-alignment story. A capacity-matched ablation (e.g., adding the same number of parameters without the context-refinement path) would be needed to support the claim that the gain comes from CLIP semantics rather than from extra parameters. Without that, the central mechanism claim is insecure.\n\nOther weaknesses: numbers appear to be single-run (no variance reported), and the paper omits RePro and UASAN results on VidOR due to unavailable code—fair, but it weakens the SOTA claim on that dataset. More importantly, METOR's VidOR all-split R@100 is actually slightly below EOV-MMP (9.72 vs 9.82), so \"state-of-the-art\" is only true on mAP and mAPo, and the recall gains are tiny (+0.11 on R@50 for VidOR all-split). That's a moderate caveat, not a fatal one. Also, hyperparameters like Ni and alpha appear to be chosen per dataset, possibly via test-set performance; that's common in the subfield but should be acknowledged. The strongest baseline is the authors' own EOV-MMP—fine in principle, but the comparison is against an incremental sibling rather than an independently implemented system, so the reader should keep that context in mind.\n\nIn short: the paper deserves a serious referee, and the empirical improvements are probably real in the narrow sense. But the causal story—that the contextual refinement improves open-vocabulary generalization through CLIP semantics—is not yet established. With a capacity-matched ablation and an independent probe of what Co/Cr encode, this could become a solid contribution. As is, I'd treat it as a useful extension rather than a conceptual breakthrough. I'd cite it if I worked on VidVRD baselines, but I wouldn't change my own approach based on the mechanism alone.\n\nMy recommendation: send it to peer review, but require a response to the capacity-confusion concern and ideally a couple of extra seeds for the main numbers.","headline":"A solid, incremental advance in Open-VidVRD with real gains on mAP and mAPo, but the central mechanism claim is undercut by a self-referential contextual loss and capacity-matched ablations are missing.","tokens_in":12410,"tokens_out":3540,"would_cite":true,"duration_ms":33961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that jointly modeling object detection and relationship classification in a single query-based framework, with context-refined text encoding and alternating mutual refinement of their representations, reduces error…","keywords":["open-vocabulary video visual relationship detection","query-based unified framework","CLIP contextual refinement","iterative enhancement","mutual enhancement of objects and relationships","scene graph detection","VidVRD","VidOR"],"falsifier":"A concrete test is to train a capacity-matched METOR variant with the context tokens, mapping layers, and contextual losses removed but with an equal number of parameters added to the transformer decoder, and compare novel-split mAP on VidVRD. If the stripped variant matches or beats METOR, the claim that CLIP context refinement is the source of generalization would be falsified.","tokens_in":11385,"feed_emoji":"🎬","tokens_out":9821,"duration_ms":91188,"temperature":0.7,"pith_summary":"Open-vocabulary video visual relationship detection asks a model to find objects and their relationships in videos even when the categories were unseen in training. The paper argues that the usual cascade, detecting objects first and then classifying relationships from those detections, lets object errors contaminate relationship predictions and that this error propagation is a major cause of failure. It proposes METOR, a single query-based model in which object detection and relationship classification share one transformer decoding stage and then refine each other alternately through several iterations. A pre-trained vision-language model (CLIP) is frozen, but learnable object and relationship context tokens are added so that both the text features and the object queries are adjusted to the video's actual content, which the paper credits with better generalization to novel categories. On the scene-graph-detection task, METOR reports the best mAP among compared methods on both VidVRD and VidOR, with the largest relative gains on novel-split categories.","feed_headline":"Joint modeling beats cascades for open-vocab video relations","feed_subtitle":"METOR alternates relationship and object updates in one model and reports top mAP on VidVRD and VidOR.","key_machinery":"The load-bearing mechanism is the contextual refinement encoding module: learnable object and relationship context tokens are concatenated with the video patch tokens and the CLS token into the fixed CLIP ViT encoder, so the resulting context embeddings $C_o$ and $C_r$ carry video-specific object and relationship context. These embeddings refine object queries through multi-head attention and, after MLP mapping layers, are prepended to category embeddings and run through the fixed CLIP text encoder to produce instance-conditioned text features. The second mechanism is the iterative enhancement module: each layer computes relationship features with a spatio-temporal transformer and then updates the subject and object features by $\\hat{O}_s^{(k)} = \\alpha \\hat{O}_s^{(k-1)} + (1-\\alpha) M_f^{(k)}(\\hat{R}^{(k)})$, with the same update for the object. This alternating loop is what operationalizes mutual enhancement and is what the ablations isolate.","core_discovery":"METOR's central claim is that objects and relationships should be recognized as one interdependent prediction problem: relationship evidence should refine object features, and refined objects should sharpen relationship features, instead of relationship prediction helplessly inheriting object mistakes. Concretely, the model appends learnable context tokens $c_o$ and $c_r$ to the input of a frozen CLIP visual encoder, producing context embeddings $C_o$ and $C_r$ that are used to refine object queries via attention and to condition the CLIP text encoder's category descriptions. An iterative enhancement module then alternates, for $N_i$ layers, between building relationship features from subject and object features through a spatio-temporal transformer and updating subject and object features from those relationship features through a mapping layer. Training is end-to-end with contrastive, trajectory, and contextual losses, and the reported evaluations show METOR ahead of prior open-vocabulary methods on SGDet mAP and object-trajectory mAP, especially for novel relationship categories.","pith_inferences":["Beyond the paper's experiments, a capacity-matched control that removes the context tokens and mapping layers but adds an equivalent number of decoder parameters would separate the CLIP-semantics effect from the effect of simply having more trainable parameters.","The alternating object-relationship refinement is not specific to video or to CLIP; the same loop could be tried for image scene graph generation, where cascade error propagation is also a known bottleneck.","The frame-level contextual losses supervise category presence from global context embeddings, so the framework could in principle be trained on weakly labeled web video with free-text captions rather than full relationship annotations."],"forward_implications":["Because trajectories and relationship labels come from the same query-based model, open-vocabulary video relation detection no longer depends on a separate close-set trajectory detector.","Relationship evidence is given a direct path back into object representations, so object errors can be corrected by relationship context rather than merely propagated forward.","The larger relative gains on novel-split evaluation indicate that context-conditioned text features help categories that were never seen during training.","The iteration study suggests that the mutual enhancement saturates quickly: two layers on VidVRD and three on VidOR are best, with extra layers slightly hurting."],"supporting_citations":[{"why":"Supplies the frozen CLIP visual and text encoders whose semantic priors the framework adapts for open-vocabulary categories.","marker":"[Radford et al., 2021]"},{"why":"Defines the Open-VidVRD base and novel split evaluation protocol and provides the RePro baseline METOR compares against.","marker":"[Gao et al., 2023]"},{"why":"Provides the OV-MMP multi-modal prompting baseline and the trajectory-detector-based cascade that motivates the unified design.","marker":"[Yang et al., 2024]"},{"why":"Provides EOV-MMP, the closest end-to-end baseline, which adds relationship context as an auxiliary loss but keeps a cascade, the design METOR improves on.","marker":"[Wang et al., 2025]"},{"why":"Provides the UASAN semantic-alignment baseline for open-vocabulary video scene graphs.","marker":"[Wu et al., 2024a]"},{"why":"Contributes the VidVRD dataset, the SGDet evaluation task, and the greedy relation association algorithm used for merging triplets.","marker":"[Shang et al., 2017]"},{"why":"Contributes the VidOR dataset used for the second benchmark and its validation set for evaluation.","marker":"[Shang et al., 2019]"},{"why":"Supplies the MS-COCO pretrained weights used to initialize the transformer decoder, excluding novel object categories.","marker":"[Lin et al., 2014]"}],"fun_headline_variants":["Mutual object-relationship enhancement tops open-vocab video VDR","One model for objects and relations beats cascades in open-vocab video","Iterative object-relationship refinement hits SOTA on VidVRD and VidOR","METOR jointly enhances objects and relationships, no more error cascades"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that its jointly trained context tokens make the pre-trained vision-language encoder's embeddings carry real information about which object and relationship categories are present, and that this contextual signal, not the extra trainable parameters, is what drives the open-vocabulary gains.","fun_headline_variants_meta":{"raw":{"variants":["Mutual object-relationship enhancement tops open-vocab video VDR","One model for objects and relations beats cascades in open-vocab video","Iterative object-relationship refinement hits SOTA on VidVRD and VidOR","METOR jointly enhances objects and relationships, no more error cascades"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2668,"prompt_tokens":944,"completion_tokens":1724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1643}},"tokens_in":560,"tokens_out":1724,"duration_ms":12999,"temperature":1.0,"reasoning_tokens":1643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:36:43.212145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test is to train a capacity-matched METOR variant with the context tokens, mapping layers, and contextual losses removed but with an equal number of parameters added to the transformer decoder, and compare novel-split mAP on VidVRD. If the stripped variant matches or beats METOR, the claim that CLIP context refinement is the source of generalization would be falsified.","supporting_citations":[{"cited_title":"Multi-modal prompting for open- vocabulary video visual relationship detection","cited_arxiv_id":null,"evidence_quote":"Provides the OV-MMP multi-modal prompting baseline and the trajectory-detector-based cascade that motivates the unified design."},{"cited_title":"End-to-end open-vocabulary video vi- sual relationship detection using multi-modal prompting","cited_arxiv_id":null,"evidence_quote":"Provides EOV-MMP, the closest end-to-end baseline, which adds relationship context as an auxiliary loss but keeps a cascade, the design METOR improves on."},{"cited_title":"Video visual relation detection","cited_arxiv_id":null,"evidence_quote":"Contributes the VidVRD dataset, the SGDet evaluation task, and the greedy relation association algorithm used for merging triplets."},{"cited_title":"Relation understanding in videos: A grand challenge overview","cited_arxiv_id":null,"evidence_quote":"Contributes the VidOR dataset used for the second benchmark and its validation set for evaluation."},{"cited_title":"Microsoft coco: Com- mon objects in context","cited_arxiv_id":null,"evidence_quote":"Supplies the MS-COCO pretrained weights used to initialize the transformer decoder, excluding novel object categories."}],"review_version":1}