{"id":"1c98242f-4062-47df-950c-1575afbc72b2","arxiv_id":"2411.16156","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Encoding video as a small set of object tokens, produced by off-the-shelf detection, segmentation, and tracking models, improves video QA accuracy and enables video-based referring in a 7B video-LLM.","lead":"VideoOrion is a video-LLM that adds a second stream of tokens representing the objects detected, segmented, and tracked in a video, alongside the usual video-context tokens. On five video QA benchmarks and a referring benchmark, the object tokens improve accuracy and enable referring by object, though the main comparisons mix in extra training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains over VideoLLaMA2/2.1 are confounded by extra training data; the controlled ablation (Table 4) uses a much weaker baseline, so the object-token contribution is not established.","rationale":"The paper's central claim is that object tokens encoding spatial-temporal dynamics are broadly useful. What must be true is that the observed gains are caused by the object branch, not by other training differences. Section 4.2's comparison does not satisfy this: VideoOrion uses ~4.4M samples including 1M ego-centric and 2.7M MM instruction data, whereas VideoLLaMA2/2.1 use their own data. The ablation (Table 4) tries to control data amount but its video-only baseline scores far below published VideoLLaMA2 (41.9 vs 53.4 MVBench), so it does not represent the same base model used in the headline comparison. The controlled gains are only ~2 points average versus 10+ points in Table 2, and Figure 8 shows VideoOrion's performance rises sharply with more training data, so most of the headline margin could be a data effect. The reader's pipeline-reliability concern is secondary: even with noisy masks, the object branch provides a useful inductive bias or extra temporal frames; but if the gains vanish under matched data, the central claim is false. The appropriate verdict remains CONDITIONAL, now specifically conditioned on a matched-data, matched-base comparison. I therefore keep the reader's verdict (UNCHANGED) while sharpening the condition.","tokens_in":18511,"tokens_out":9975,"duration_ms":81092,"concrete_test":"Train a video-only (no Object-Centric Branch) model on exactly the same data mixture as VideoOrion (Table 1): 700K object-pretraining videos as regular video-text pretraining, 2.7M MM video-text, 1M ego-centric, 625K image-text, 40K text-only, initialized from the same VideoLLaMA2 STC connector, with identical hyperparameters (Table 13) and token budget. Evaluate it on MVBench, EgoSchema, Perception-Test, VideoMME, and ActivityNet-QA. If it reaches VideoOrion's scores within ~3 points, the headline gains are not attributable to object tokens; if it lags by the 5–15 point margins, the concern is refuted. A cheaper intermediate check is to verify that the Table 4 video-only baseline reproduces published VideoLLaMA2 numbers (e.g., 53.4 MVBench); if it does not, the ablation baseline is too weak to draw causal conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §4.2 is that the Object-Centric Branch makes VideoOrion 'consistently surpass' VideoLLaMA2 and VideoLLaMA2.1 by 5.1–15.6 points. This comparison is not controlled: VideoOrion is trained on a larger, different data mixture (Table 1: +1M ego-centric QA, +2.7M MM instruction tuning, +700K object-pretraining videos), while VideoLLaMA2/2.1 are evaluated with their original training recipes. The only data-controlled ablation (Table 4) trains a video-only baseline on 702K Valley + 558K LLaVA + 765K Video-LLaVA, with the 700K object-pretraining videos added as video-text pretraining, yielding 41.9 on MVBench versus the published VideoLLaMA2's 53.4. Thus the ablation baseline is much weaker than the base model used in the headline comparison, so it cannot isolate the object-token effect. The controlled gains in Table 4 are only ~2 points average (44.2 vs 41.9 on MVBench), an order of magnitude smaller than the 10+ point headline margins, leaving open that the Table 2 gains come from data scale/base strength, not object tokens. Without a matched-data, matched-base comparison, the causal claim that object tokens improve video understanding is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VideoOrion is a video-LLM with two encoding branches: a standard video-centric branch using an STC connector, and an object-centric branch that generates object masks via a detect-segment-track pipeline (GroundingDINO, SAM, RAM++, XMem), pools mask-aligned CLIP features, and maps them to object tokens. The model is trained in three stages (video projector pretraining, object projector pretraining on 700K filtered videos, and multi-modal instruction tuning including 1M ego-centric QA samples). Evaluations on MVBench, EgoSchema, Perception-Test, VideoMME, ActivityNet-QA, and video referring (VideoRef45K) show large gains over VideoLLaMA2/2.1 and competitive or superior performance on referring. Ablations examine the object branch, pipeline design choices, projector architecture, and token counts.","tokens_in":54,"tokens_out":5921,"duration_ms":94063,"significance":"The idea of explicitly encoding object dynamics as compact, semantically meaningful tokens is a promising addition to video-LLM design. The paper contributes a concrete pipeline and extensive ablations, including design-choice analysis, failure cases, and data-scaling behavior. The controlled ablation in Table 4 shows consistent improvements over a same-data video-only baseline, albeit of modest size, supporting a weaker version of the central claim. However, the headline comparison in Table 2 is not controlled for training data or base-model strength, so the significance of the object-token contribution depends on whether the large margins reflect the object branch or simply a larger/stronger training setup.","major_comments":[{"comment":"The claim that VideoOrion 'consistently surpasses' VideoLLaMA2 and VideoLLaMA2.1 by 5.1–15.6 points is not supported as a causal statement about the Object-Centric Branch, because the comparison is not matched on training data. Table 1 shows VideoOrion is trained with 700K object-pretraining videos, 1M ego-centric QA samples, and 2.7M MM instruction-tuning samples, whereas VideoLLaMA2/2.1 use their original training recipes. The only data-controlled ablation (Table 4) yields gains of about 2 points average (e.g., MVBench 44.2 vs 41.9), an order of magnitude smaller than the headline margins. Please provide a matched-data, matched-base comparison (e.g., train VideoLLaMA2 on the same mixture, or ablate object tokens on the full model) or substantially qualify the claim in §4.2.","section":"§4.2, Table 2"},{"comment":"The data-controlled baseline in Table 4 is substantially weaker than the published VideoLLaMA2 (MVBench 41.9 vs 53.4), suggesting that the controlled comparison is performed on a limited-capacity training configuration. It is unclear whether the small object-token gains would persist with the full training scale. Please report a full-scale ablation or explain why the subset results are representative, and specify the exact composition of the video-only baseline (e.g., whether the 700K video data are indeed added to the video pretraining stage for the baseline).","section":"Table 4"},{"comment":"No error bars or repeated-trial variance are reported for any table. The differences in Table 4 are only 1–2 points on some benchmarks (e.g., ActivityNet-QA 43.3 vs 43.0), so it is difficult to assess whether the 'consistent' improvements are beyond evaluation noise. Please provide multiple seeds or an estimate of benchmark variability, and explain why Table 7 reports different VideoOrion numbers (MVBench 48.0) than Table 4 (MVBench 44.2) for what appears to be the same configuration.","section":"§4.4, Tables 4 and 7"}],"minor_comments":[{"comment":"The abstract and introduction describe the method as having 'minimal computational cost,' but Appendix E reports a 38.5% increase in per-sample time; please soften the phrasing to 'moderate additional cost.'","section":"Abstract and Appendix E"},{"comment":"The sentence 'we sample to frames from the video' uses the variable 'to' before it is defined; introduce the symbol (e.g., to) with its meaning and value before this sentence.","section":"§3.3"},{"comment":"The phrase 'which has the same Video-Centric Branch' should be 'which have the same Video-Centric Branch' to agree with the plural subject 'VideoLLaMA2 and VideoLLaMA2.1.'","section":"§4.2"},{"comment":"The ActivityNet-QA column mixes formats such as '53.5 /3.2' and '57.7/3.7'; use consistent spacing and a clear 'Accuracy / Score' header.","section":"Table 2"},{"comment":"The comparison with VideoMAE and UMT-L is useful, but the text says 'more than 2% on average' while the table shows 45.3 vs 43.0 average; please check the arithmetic and state the exact average gain.","section":"§4.4, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant problem and the experimental apparatus is extensive, but the evaluation needs to be tightened before the central claim can be accepted. The data-confounding issue is the primary concern; the controlled ablation, though directionally consistent, is too small and too weak a baseline to justify the headline margins. I am not recommending rejection because the object-token idea is plausible and the ablation provides some support, but a major revision with a matched-data comparison at full scale is necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the object-token idea is genuinely new, but the headline claim in §4.2 is not supported by the experiments as written. VideoOrion adds a detect-segment-track branch (GroundingDINO → SAM → XMem, with RAM++ picking key frames) that produces object tokens fed to the LLM alongside standard context tokens. That is a real departure from Slot-VLM's slot attention and Artemis's referring-only focus. The referring results (Table 3) are strong, and the ablation suite (Tables 5–11) is unusually thorough: pipeline variants, object projector choices, comparison against video encoders like VideoMAE and UMT-L, token-count limits, and a careful failure analysis. Credit where due—this is a solid method-level contribution, and the paper is honest that the pipeline depends on upstream mask quality (Limitations, Appendix D.1).\n\nThe soft spot is the evaluation. The §4.2 claim of 'consistently surpassing' VideoLLaMA2/2.1 by 5–15 points compares against the published baselines, while VideoOrion trains on far more data: 1M ego-centric QA, 2.7M instruction tuning, and 700K object pretraining. So the margin could come from data scale, not from object tokens. The controlled ablation in Table 4 is data-matched, but the base there is much weaker than the published VideoLLaMA2 (41.9 vs 53.4 on MVBench), and the object-token gain is only about 2 points on average. That does not tell you what object tokens would add to a strong base. Without a matched-base experiment—object branch on the full VideoLLaMA2 recipe with identical data—the causal claim is unverified. No error bars, no multiple seeds, and no code/data release make independent checks impossible. The free parameters (θa=3, θb=2, 64-token cap) are tuned on the ablation set; minor, not fatal.\n\nWho this is for: researchers working on video-LLM representations will get real value from the object-token formulation and the ablation methodology. It deserves a serious referee—the paper is important enough and the contributing components are well-constructed. The referee should insist on a clean matched-base comparison and artifact release before any accept. As is, a conditional accept with major revision.","headline":"The object-token idea is worth serious attention, but the headline gains are confounded by extra training data and a weak controlled baseline.","tokens_in":19372,"tokens_out":5353,"would_cite":true,"duration_ms":45655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that encoding each tracked object's spatial-temporal dynamics as a single object token gives video-language models a compact, disentangled visual representation that improves general question answering and makes…","keywords":["video large language models","object-centric representation","detect-segment-track pipeline","object tokens","video question answering","video-based referring","spatial-temporal object dynamics","video tokenization"],"falsifier":"A mask-corruption control would settle the claim: run VideoOrion with the same number and size of object tokens, but pool features from random or shuffled regions instead of the detected masks. If MVBench and VideoMME accuracy stay within a point or two, the object-token mechanism is not doing the claimed work; if accuracy drops sharply, the tracked masks are load-bearing.","tokens_in":18371,"feed_emoji":"🎬","tokens_out":6086,"duration_ms":55342,"temperature":0.7,"pith_summary":"The paper is trying to establish that object-centric tokens—one token per tracked object, pooling its masked features across frames—are a better visual vocabulary for video-language models than downsampled patch tokens. It argues that this representation is more compact, more disentangled, and more aligned with text semantics, and it claims that adding such an Object-Centric Branch to a standard frame-based Video-LLM yields consistent gains on five video question-answering benchmarks and large gains on video-based referring. If right, this means a video-LLM does not have to choose between coarse frame sampling and expensive full-video tokenization; object dynamics can be carried in a handful of extra tokens.","feed_headline":"A token per tracked object lifts video QA over frame-only models","feed_subtitle":"VideoOrion beats its own frame-only baselines on five benchmarks and gains object-level referring.","key_machinery":"The central object is the object token: one vector in the LLM embedding space that summarizes one object's appearance and motion across the whole video. It is produced by a detect-segment-track pipeline—GroundingDINO proposes boxes, SAM segments masks, RAM++ selects key frames by object-tag overlap, and XMem tracks each mask across up to 128 frames—followed by an Object Projector that mask-pools CLIP features per frame and temporal-pools those pooled features before passing them through an MLP. The Object-Centric Branch thus compresses each object's spatial-temporal dynamics into one token while the Video-Centric Branch supplies context tokens.","core_discovery":"The central claim is that explicitly tokenizing objects—their masks, trajectories, and feature dynamics—gives a Video-LLM a compact, disentangled semantic representation that downsampled frame patches cannot match. Concretely, adding the Object-Centric Branch to VideoLLaMA2 and VideoLLaMA2.1 makes the model consistently surpass those baselines by 5.1 to 15.6 points across MVBench, EgoSchema, Perception-Test, VideoMME, and ActivityNet-QA, and turns the same interface into a video-referring system that outperforms Artemis on VideoRef45K. The author's own framing is that the object tokens make video representation more like text tokenization, where each token already carries clear semantics.","pith_inferences":["A natural extension is to use the same object-token interface for tasks that require object identity rather than open-ended description—such as video grounding, editing, or embodied decision-making—since the representation already separates instances.","The design suggests that long-video understanding could be improved by scaling the tracking branch with more frames or faster trackers rather than by lengthening the LLM's context, a cheaper direction than long-context video transformers.","Because the object branch sees 64 to 128 frames while the video branch sees only 8 to 16, a control that gives the video branch the same extra frames would clarify whether the gains come from disentangled semantics or simply from richer temporal sampling.","The paper's failure analysis implies object tokens also make errors inspectable; a future system could add a per-token verification step that checks whether mask pooling is actually attending to the intended object."],"forward_implications":["General video QA improves on all five tested benchmarks, with same-backbone gains of 5.1 to 15.6 accuracy points, even though the video branch still sees only 8 to 16 frames.","Video-based referring becomes a natural capability: inserting the target object's token into the prompt outperforms dedicated referring models such as Artemis on VideoRef45K across all reported metrics.","Fine-grained temporal reasoning improves: VideoOrion+ scores 61.8 versus 59.5 multi-binary accuracy on TemporalBench compared with VideoLLaMA2.1.","Replacing the object branch with standard video encoders such as VideoMAE or UMT-L costs more than 2 average points, indicating the object representations, not just extra temporal features, drive the gain.","Object-branch pretraining on 700K video-text pairs is necessary; without it most gains disappear, and object-only encoding is not enough on its own, so both branches are needed."],"supporting_citations":[{"why":"GroundingDINO supplies the open-vocabulary object proposals that start every object token.","marker":"[38]"},{"why":"SAM converts detected boxes into precise masks used for mask pooling.","marker":"[27]"},{"why":"XMem tracks each mask across frames so one object token aggregates a single instance's dynamics.","marker":"[13]"},{"why":"RAM++ tags frames and drives the key-frame segmentation that decides when new objects appear.","marker":"[23]"},{"why":"VideoLLaMA2 provides the video branch, the STC connector, and the direct same-backbone baseline the object tokens must beat.","marker":"[14]"},{"why":"Artemis gives the video-referring benchmark VideoRef45K and the finetuning protocol against which the referring results are measured.","marker":"[51]"},{"why":"CLIP is the vision encoder that extracts the frame features both branches pool into tokens.","marker":"[52]"}],"fun_headline_variants":["Object tokens beat downsampled frames in video QA by up to 15.6 points","VideoOrion tokenizes object dynamics for compact, disentangled video semantics","Object tokens: a new way to compress video for LLMs without losing semantics","Track objects, tokenize them, and video QA sees a 15.6-point boost","Object dynamics as tokens make video LLMs sharper and more referable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The object-token benefit rests on the upstream pipeline reliably producing the right mask for the right object in every frame; if detection, key-frame selection, or tracking fails, the added tokens carry noise rather than disentangled object semantics.","fun_headline_variants_meta":{"raw":{"variants":["Object tokens beat downsampled frames in video QA by up to 15.6 points","VideoOrion tokenizes object dynamics for compact, disentangled video semantics","Object tokens: a new way to compress video for LLMs without losing semantics","Track objects, tokenize them, and video QA sees a 15.6-point boost","Object dynamics as tokens make video LLMs sharper and more referable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2779,"prompt_tokens":880,"completion_tokens":1899,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1795}},"tokens_in":496,"tokens_out":1899,"duration_ms":12610,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:28:52.163970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A mask-corruption control would settle the claim: run VideoOrion with the same number and size of object tokens, but pool features from random or shuffled regions instead of the detected masks. If MVBench and VideoMME accuracy stay within a point or two, the object-token mechanism is not doing the claimed work; if accuracy drops sharply, the tracked masks are load-bearing.","supporting_citations":[{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollár, and Ross B","cited_arxiv_id":null,"evidence_quote":"SAM converts detected boxes into precise masks used for mask pooling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"XMem tracks each mask across frames so one object token aggregates a single instance's dynamics."},{"cited_title":"Open-set image tagging with multi-grained text su- pervision","cited_arxiv_id":null,"evidence_quote":"RAM++ tags frames and drives the key-frame segmentation that decides when new objects appear."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"CLIP is the vision encoder that extracts the frame features both branches pool into tokens."}],"review_version":1}