Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The paper claims that a 4B-parameter video language model gains space-time referring and reasoning abilities from synthetic instruction data produced by an open-source data engine.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:52 UTC pith:YADIXKGK

load-bearing objection Useful open-source data engine; mask-referring gains are credible, but timestamp gains are confounded with new architecture. the 4 major comments →

arxiv 2509.03501 v1 pith:YADIXKGK submitted 2025-09-03 cs.CV cs.AIcs.HCcs.LG

Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data

Honglu Zhou , Xiangyu Peng , Shrikant Kendre , Michael S. Ryoo , Silvio Savarese , Caiming Xiong , Juan Carlos Niebles This is my paper
classification cs.CV cs.AIcs.HCcs.LG
keywords space-time referringVideo LLMssynthetic instruction datamasklettemporal reasoningpseudo-annotationinstruction tuningvideo question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that Video LLMs can learn to answer queries that point at a specific object and a specific moment in a video—what it calls space-time referring—using instruction data synthesized entirely from frozen open-source models, with no proprietary models or human annotation. Strefer first pseudo-annotates short videos with dense metadata: which entities are active, where each one is as a tracked mask (a masklet), and what each one is doing during each clip. It then converts that metadata into roughly 950,000 instruction-response pairs whose questions mix masks and timestamps. Trained on the base recipe plus this data, a 4B Video LLM improves on mask-referred description and QA, timestamp-referred QA, and temporal reasoning benchmarks compared with the same model trained without Strefer data. If the claim holds, this is a scalable, low-cost route to perceptually grounded video instruction data.

Core claim

Strefer's central discovery is that a space-time referring task—requiring the model to use both a user-specified region and a user-specified time interval—can be taught entirely with synthetically generated QA pairs, and that doing so sharpens general video temporal reasoning. The paper demonstrates this by taking a fixed 4B Video LLM architecture, keeping the base recipe, and adding Strefer's synthesized data groups one at a time. The final recipe raises VideoRefer-BenchD average from 3.2837 to 3.3947, VideoRefer-BenchQ from 0.665 to 0.688, QVHighlights timestamp Yes/No from 0.5288 to 0.6031, TempCompass from 60.100 to 61.675, and VideoMME temporal subsets from 37.45 to 37.70. The strongest

What carries the argument

The load-bearing object is the masklet: a segmentation mask of a single entity tracked over time, paired with a natural-language referring expression. Strefer's engine builds it in four steps: sample and reorder frames to find a tracking-initialization frame where all entities are visible; detect objects using short generalized nouns; track them both forward and backward in time; and assign each multi-word referring expression to a masklet. A Video LLM writes per-clip presence checks and behavior descriptions, and an LLM turns the resulting timeline into template-based or open-ended QA pairs. On the model side, two plug-and-play modules—a region-language connector that pools visual features

Load-bearing premise

The entire approach depends on the frozen open-source models used to detect, track, and describe entities being accurate enough that the synthesized questions and answers are genuinely about what appears in the video; if those pseudo-annotations contain many errors, the training data may teach hallucination rather than grounding.

What would settle it

Have humans verify a random sample of Strefer's synthesized QA pairs against the source videos, counting how often the masklet points at the wrong entity or the claimed action occurs outside the stated timestamps; then retrain the final recipe on only the verified-correct subset. If the gain over baseline does not survive or shrinks, the claim that grounded synthetic data is causally responsible for the improvement is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Space-time referring supervision transfers: short mask-and-timestamp QA data also improves performance on TempCompass and VideoMME, benchmarks not built around masks.
  • Data quality beats scale: adding 27K well-formed mask-plus-timestamp questions to a 1.9M-sample base recipe moves several benchmarks, while a 1K event-ordering group can hurt fine-grained tasks.
  • Mask-referred and timestamp-referred data are complementary: mask-only data helps subject correspondence, timestamp-only data helps temporal QA, and their combination delivers the best overall recipe.
  • Negative and presence questions help: template questions that ask about absent entities or respond with uncertainty improve both temporal and referring benchmarks.
  • Visual prompting without retraining gives limited gains; overlaying masks or frame numbers on frames is not enough until combined with model tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the engine is modular and uses only open-source components, its output quality is bounded by its weakest component; the documented failure modes (motion blur, occluded entities, mask gaps) are places where a stronger tracker or captioner would most directly raise downstream benchmark scores.
  • Since the pipeline consumes only raw videos, it can in principle be pointed at any unlabeled corpus—egocentric, surveillance, or long-form—to generate region-and-time instruction data, which the paper motivates but does not test.
  • The QVHighlights result uses a repurposed Yes/No benchmark constructed by the authors, so its gains should be re-checked on an existing timestamp-grounding benchmark before attributing them mainly to the temporal-token module.
  • A natural next experiment is to train on full masklets instead of one sampled mask per instruction; the paper notes this is expected to improve performance further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Strefer, a modular synthetic instruction-data engine that pseudo-annotates videos with object-centric, space-time metadata—active entities, masklets, action descriptions, and timelines—using only frozen open-source models (Tarsier-34B, Qwen2.5-32B, GroundingDINO, SAM2, RexSeek). From 4,253 NExT-QA videos it generates 947,854 instruction QA pairs grouped into eight ablations (G1–G8), covering mask-referring, timestamp-referring, and traditional temporal tasks. The authors train a 4B BLIP-3-Video-based model with added region-language and timestamp-token modules, using a base recipe of BLIP-3-Video data plus VideoRefer-700K. They report that adding Strefer groups improves mask-referred regional description/QA on VideoRefer-BenchD/Q, a newly constructed timestamp Yes/No task on QVHighlights, TempCompass, and VideoMME temporal subsets. The central claim is that Strefer data, without proprietary models or manual annotation, improves space-time referring and reasoning in Video LLMs.

Significance. If the empirical claims hold, Strefer is a valuable contribution: it provides a scalable, open-source recipe for synthesizing fine-grained spatiotemporal instruction data, handles challenging scenarios such as multiple same-category entities and entities that exit/re-enter, and publishes extensive ablations (Tables 2–4) plus qualitative analyses. The paper is also unusually candid about failure cases and limitations (Sec. 5). The strongest evidence is the consistent improvement on VideoRefer-BenchD/Q, where the base recipe already includes mask-referring data and the region connector, making the additive effect of Strefer data more interpretable. However, the timestamp-referring result is presently confounded with architectural changes, and the main timestamp benchmark is self-constructed, so the central claim needs additional controls before it can be accepted at face value.

major comments (4)
  1. [§3.1, Table 4] The QVHighlights gain (0.5288 → 0.6031) is confounded with architecture. The Baseline explicitly 'does not have Timestamp Conversion or an extended LLM vocabulary for learning special temporal tokens' and omits modules when training data lacks the corresponding instructions. Adding G7/G8 simultaneously introduces Strefer's timestamp QA and these temporal-token modules. Thus Finding 1's attribution of the improvement to Strefer data is not identifiable. A same-architecture control—e.g., the base recipe plus a non-Strefer timestamp instruction source, or a Strefer-data variant without temporal-token modules—is required before the data can be called causal.
  2. [§3.2, Table 4] The timestamp-referred QA evaluation on QVHighlights is a newly constructed benchmark, not an established task. Its question template ('Does the following description accurately reflect what happens in the video between <start> and <end>?') closely matches Strefer's timestamp-referring instruction formats, and the trained model receives temporal tokens at both training and eval. I do not see label leakage—QVHighlights annotations are not used in training—but the format and token representation are aligned by construction. The timestamp claim should be validated on an external, independently designed temporal grounding/QA benchmark (e.g., Charades-STA or ActivityNet-Captions grounding) before it is used as a headline result.
  3. [Tables 2–4] No error bars, significance tests, or multiple seeds are reported. Several headline deltas are small (TempCompass 60.100 → 61.675; VideoMME 37.45 → 37.70; BenchD avg 3.2837 → 3.3947), and the ablations are non-monotonic: adding G2 alone raises VideoMME to 41.65, but the final G1+G2 recipe drops to 34.70. Without variance estimates, the claim that the final recipe 'consistently and significantly improves performance' is not supported. At minimum, the key comparisons (Baseline vs. final recipe, Baseline vs. +G7) should be run with at least 3 seeds or bootstrap confidence intervals.
  4. [§5, Fig. 8, Fig. 11, Fig. 2] The paper's causal story is that Strefer data improves grounding because it is grounded in spatiotemporal metadata. Yet the manuscript provides no quantitative measure of pseudo-annotation quality—e.g., masklet IoU vs. manual masks, presence-check accuracy, or human ratings of synthesized QA correctness. The cited failure cases (occluded woman declared absent, child masks missing in frames 10–11, motion-blurred bicycle track) are acknowledged but not quantified. A small human-annotated sample or a filtering-based quality analysis would directly test the assumption that the data engine is accurate enough to teach grounding rather than hallucination. This is a correctness-risk concern, not a circularity claim.
minor comments (5)
  1. [Multiple figure captions (Fig. 1, 2, 3, 9, 10, 21)] Typos and grammar issues: 'does not any use', 'Synthethized', 'hoddie', 'meting'. Please proofread.
  2. [Fig. 6] The percentages and sample counts do not obviously sum to the stated totals. Clarify the denominator and whether the figure refers to the final training mixture or all generated groups.
  3. [Table 4] The VideoMME column header is ambiguous (Avg. Yes/No MCQ Caption Matching Captioning). Please define exactly which VideoMME subsets and averaging are used; the caption says 'Temporal Perception and Temporal Reasoning subsets (no subtitles)', but the table layout is unclear.
  4. [§3.1] Training samples a single mask on a random frame per instruction pair, while evaluation uses the full masklet. This train/eval mismatch is a potentially important detail; please discuss its effect or provide an ablation with full-masklet training.
  5. [§3.1] The names 'Baseline' and 'Baseline Ablation' are easy to confuse. Consider renaming to 'Base Recipe' and 'Image/Video-Only Recipe' for clarity.

Circularity Check

0 steps flagged

No material circularity: the reported gains are evaluated on external annotations; the main caveats (self-cited base recipe, QVHighlights template overlap, architecture confound) affect attribution, not derivation-by-construction.

full rationale

Strefer's synthesis pipeline (Sec. 2.1) produces training QA pairs from pseudo-annotated metadata over NExT-QA videos, while the reported benchmarks use external labels not used to generate training data: VideoRefer-BenchD/Q (Panda-70M test; MeViS/A2D/Refer-YouTube-VOS test), TempCompass, VideoMME, and QVHighlights human segment annotations (Sec. 3.2, Sec. 4.1.1). No equation or fitted parameter is reused as a prediction; the training/evaluation inputs are disjoint. The QVHighlights Yes/No task is constructed by the authors but its answers come from human highlight annotations and non-overlapping negative windows, so the benchmark score is not determined by the Strefer training data. The paper's self-citation of BLIP-3-Video [45] as the base recipe is not load-bearing for the claim that Strefer data adds value; it is an external public baseline. Section 3.1 does reveal an identifiability caveat: timestamp-related modules are omitted when timestamp-referring instructions are absent, so the QVHighlights improvement with G7/G8 changes architecture and data simultaneously. This is a potential confound for causal attribution, not a circular reduction. Section 5 openly acknowledges pseudo-annotation errors (occluded woman, missing masks, motion blur) and model limitations; these are correctness risks, not circularity. Overall, the central claim has independent empirical content.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities; the load-bearing inputs are hand-chosen pipeline thresholds and the accuracy assumptions of the frozen pretrained models. The central empirical claim depends on these rather than on a derivation.

free parameters (8)
  • PySceneDetect ContentDetector threshold = 20
    Empirically chosen based on qualitative assessment (Section 2.1.4); affects clip boundaries and thus transcript and QA content.
  • HAC auto-threshold multiplier f = 1.7
    Used in Algorithm 2 CLUSTERING AUTOTHRESHOLD to set the clustering distance threshold; controls semantic segmentation of clips.
  • SigLIP frame rate = 3 FPS
    Frame subsampling rate for semantic clip detection (Section 2.1.4).
  • QVHighlights negative buffer = 5 seconds
    Buffer added to both sides of annotated intervals when excluding intervals for negative sample generation (Section 3.2); determines benchmark difficulty.
  • QVHighlights negative minimum duration = 10 seconds
    Minimum duration for randomly selected negative segments (Section 3.2); affects the label distribution and difficulty.
  • Number of temporal tokens = 32 (M=31)
    Sets temporal granularity for timestamp tokens (Section 3.1, A.3.4); chosen to match 32 input frames.
  • Number of frames per video = 32
    Sampling rate for training and evaluation (Section 3.1).
  • Masklet frame sampling for training = 1 random frame per instruction pair
    They sample a single mask on a random frame per instruction pair, not the full masklet (Section 3.1); a design choice likely affecting performance.
axioms (5)
  • domain assumption Pseudo-annotated metadata (active entities, masklets, per-clip behavior descriptions) from Tarsier-34B, GroundingDINO, SAM2 and RexSeek is accurate enough for correct instruction-response pairs.
    The entire data engine is built on this chain; Section 5 concedes concrete failure modes (missed occluded woman, lost masks, motion blur).
  • domain assumption A translation from timestamp to 32 discrete temporal tokens preserves temporal-reference information needed for QA.
    Section A.3.4; rounding timestamps to M=31 anchors loses precision that may matter for short events.
  • domain assumption The repurposed QVHighlights Yes/No benchmark labels are correct, i.e., randomly selected non-overlapping gaps never actually match a description from an annotated segment.
    Section 3.2; if a description's content recurs in a gap (e.g., repeated 'overhead shots of breakfast'), the constructed 'No' label is wrong, affecting the reported accuracy.
  • domain assumption VideoRefer-BenchD/Q and TempCompass/VideoMME scores, including GPT-4o-as-judge scores, are valid measures of space-time referring and reasoning.
    Sections 3.2 and 4.1; GPT-4o scoring is itself an LLM judgment with known biases, and no human agreement is reported.
  • domain assumption Fine-tuning with a frozen visual encoder and untrained plug-in modules is sufficient to learn the new region and timestamp token semantics.
    Section 3.1; the visual encoder is frozen due to insufficient data, and the new modules start untrained; this may limit the gains.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 34008 in / 18096 out tokens · 170129 ms · 2026-08-05T10:52:12.099663+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data." pith.science (2026). https://pith.science/paper/YADIXKGK

@misc{pith2026250903501,
  author       = {Pith},
  title        = {Pith review of: Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YADIXKGK}},
  note         = {Machine review of arXiv:2509.03501}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Next-generation AI companions must go beyond general video understanding to resolve spatial and temporal references in dynamic, real-world environments. Existing Video Large Language Models (Video LLMs), while capable of coarse-level comprehension, struggle with fine-grained, spatiotemporal reasoning, especially when user queries rely on time-based event references for temporal anchoring, or gestural cues for spatial anchoring to clarify object references and positions. To bridge this critical gap, we introduce Strefer, a synthetic instruction data generation framework designed to equip Video LLMs with spatiotemporal referring and reasoning capabilities. Strefer produces diverse instruction-tuning data using a data engine that pseudo-annotates temporally dense, fine-grained video metadata, capturing rich spatial and temporal information in a structured manner, including subjects, objects, their locations as masklets, and their action descriptions and timelines. Our approach enhances the ability of Video LLMs to interpret spatial and temporal references, fostering more versatile, space-time-aware reasoning essential for real-world AI companions. Without using proprietary models, costly human annotation, or the need to annotate large volumes of new videos, experimental evaluations show that models trained with data produced by Strefer outperform baselines on tasks requiring spatial and temporal disambiguation. Additionally, these models exhibit enhanced space-time-aware reasoning, establishing a new foundation for perceptually grounded, instruction-tuned Video LLMs.

Figures

Figures reproduced from arXiv: 2509.03501 by Caiming Xiong, Honglu Zhou, Juan Carlos Niebles, Michael S. Ryoo, Shrikant Kendre, Silvio Savarese, Xiangyu Peng.

Figure 1
Figure 1. Figure 1: Our goal is to synthesize instruction-response pairs through a scalable, grounded approach that enhances fine-grained spatial and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example of Strefer-annotated instruction-response pairs (bottom) and video metadata (top). Each instruction begins with the prefix: Please answer the following question about the <region> (omitted in the figure). For each instruction-response pair, the boundary of object mask referred to by <region> is shown beside the pair. Strefer automatically clips the video into segments and pseudo-annotates the video… view at source ↗
Figure 3
Figure 3. Figure 3: We introduce Strefer, a novel data engine that automatically generates synthetic instruction data—without manual effort or legacy annotation—featuring multimodal prompts grounded in complex spatiotemporal video structures, designed to train Video LLMs for space-time referring and reasoning tasks (Sec. 2.1). By design, Strefer handles challenging scenarios—such as scenes containing multiple entities of the … view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the Referring Masklet Generation Pipeline within Strefer. This pipeline produces tracked segmentation masks from videos with complex structures based on multi-word natural language referring expressions. Our masklet generator is carefully crafted to address key limitations overlooked by prior works [22, 37, 68] by orchestrating complementary strengths of the state-of-the-art pixel-level vision … view at source ↗
Figure 5
Figure 5. Figure 5: Model Architecture: Plug-and-play modules (Region￾Language Connector, Timestamp Conversion) enhance general￾purpose Video LLMs with space-time referring capabilities. It is worth noting that incorporating these modules for space-time re￾ferring is not strictly necessary (see Sec. 2.3). assumptions.” The term “entity” is substituted with the actual referring expression (e.g., “girl in a white dress”). This … view at source ↗
Figure 6
Figure 6. Figure 6: Data composition of our final recipe used in our exper￾iments in Sec. 4. proaches, in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative Results of Referring Masklet Generation. In this video, our method accurately generates masklets corresponding to the input referring expressions. In contrast, GroundedSAM2 [44] fails to assign expressions to masklets and does not detect the woman and the child in a pink top, who appear midway through the video. 4.2. Qualitative Results In this section, we present qualitative results that best … view at source ↗
Figure 8
Figure 8. Figure 8: Failure Results of Referring Masklet Generation. Our method fails to consistently track the woman on a bicycle throughout the video, while GroundedSAM2 [44] fails to detect, track, and differentiate the individuals referenced in the input text prompt. Videos with heavy motion blur and long-range dependencies remain challenging to handle. 4.2.2. Our Referring Masklet Generation We present qualitative result… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchD , designed to assess a model’s performance on the task of Mask-Referred Regional Description. The boundary of the region referred to by the mask in this sample is highlighted in purple. While the video includes several individuals as prominent foreground elements, the masklet specific… view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from QVHigh￾lights, using our repurposed task designed to assess a model’s performance on Timestamp-Referred Video QA. The segment boundaries corresponding to the timestamps in the first and second questions are highlighted in purple and green, respectively. The model trained on our Strefer-generated data … view at source ↗
Figure 11
Figure 11. Figure 11: Example of Strefer-Synthesized Instruction-Response Pairs (left) and Pseudo-Annotated Video Metadata (right). Each instruction begins with the prefix: “Please answer the following question about the <region>” (and the prefix is omitted in the figure). For each instruction-response pair, the boundary of the object mask referred to by <region> is shown next to the pair and highlighted in color. Strefer auto… view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative Results of Referring Masklet Generation. In this video, our method accurately generates masklets corresponding to the input referring expressions. In contrast, GroundedSAM2 [44] fails to differentiate between the man in the grey shirt and the man in the black jacket. A.3. Model Details A.3.1. Architecture Overview The Video LLM processes a video and a user’s multimodal query to generate a text… view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative Results of Referring Masklet Generation. Our method accurately generates masklets corresponding to the input referring expressions. In contrast, GroundedSAM2 [44] converts the input text prompt into the class names groom officiant, groom, bride, and bridesmaid. GroundedSAM2 then fails to detect the bridesmaid in the video, while incorrectly assigning the class name bridesmaid to the actual bri… view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchD , designed to assess a model’s performance on the task of Mask-Referred Regional Description. The boundary of the region referred to by the mask in this sample is highlighted in purple. The baseline model tends to have a foreground bias, fails to interpret the masklet correctly, and … view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchQ , designed to assess a model’s performance on the task of Mask-Referred Regional QA. The boundary of the region referred to by the mask in this sample is highlighted in purple. The model trained on Strefer-generated data correctly identifies the masklet-referred region and action. th… view at source ↗
Figure 16
Figure 16. Figure 16: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchQ , designed to assess a model’s performance on the task of Mask-Referred Regional QA. The boundary of the region referred to by the mask in this sample is highlighted in purple. In this sample, the model must demonstrate fine-grained spatiotemporal action understanding due to the smal… view at source ↗
Figure 17
Figure 17. Figure 17: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchQ , designed to assess a model’s performance on the task of Mask-Referred Regional QA. The boundary of the region referred to by the mask in this sample is highlighted in purple. The model trained on Strefer-generated data correctly identifies the masklet-referred region and action. au… view at source ↗
Figure 18
Figure 18. Figure 18: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchQ , designed to assess a model’s performance on the task of Mask-Referred Regional QA. This sample presents a multi-masklet scenario, with two masklets referring to two different individuals. The boundary of the <object1> region is highlighted in purple, and <object2> is highlighted in… view at source ↗
Figure 19
Figure 19. Figure 19: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchQ , designed to assess a model’s performance on the task of Mask-Referred Regional QA. This sample presents a multi-masklet scenario, with two masklets referring to two different individuals. The boundary of the <object1> region is highlighted in purple, and <object2> is highlighted in… view at source ↗
Figure 20
Figure 20. Figure 20: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from QVHigh￾lights, using our repurposed task designed to assess a model’s performance on Timestamp-Referred Video QA. The boundary of segment corresponding to the timestamps in the question is highlighted in purple. The model trained on our Strefer-generated data correctly an￾swers the question, demonstra… view at source ↗
Figure 21
Figure 21. Figure 21: Qualitative Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from QVHigh￾lights, using our repurposed task designed to assess a model’s performance on Timestamp-Referred Video QA. The boundary of segment corresponding to the timestamps in the question is highlighted in purple. The model trained on our Strefer-generated data correctly an￾swers the question, demonstra… view at source ↗
Figure 22
Figure 22. Figure 22: Failure Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchD , designed to assess a model’s performance on the task of Mask-Referred Regional Description. The boundary of the region referred to by the mask in this sample is highlighted in purple. The masklet is intended to refer to the boy on the right, but he is mostly out of view, while a woman … view at source ↗
Figure 23
Figure 23. Figure 23: Failure Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from VideoRefer￾BenchD , designed to assess a model’s performance on the task of Mask-Referred Regional Description. The boundary of the region referred to by the mask in this sample is highlighted in purple. While the model trained on Strefer-generated data correctly identifies that the masklet refers specifi… view at source ↗
Figure 24
Figure 24. Figure 24: Failure Results of Video LLMs Trained w/ and w/o Strefer-Synthethized Data. This sample is drawn from QVHighlights, using our repurposed task designed to assess a model’s performance on Timestamp-Referred Video QA. The boundary of segment corresponding to the timestamps in the question is highlighted in purple. Although the model trained on our Strefer-generated data fails to answer the question correctly… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly

    cs.CV 2026-05 unverdicted novelty 6.0

    Flat-Pack Bench is a new evaluation suite that shows state-of-the-art LVLMs perform poorly on nuanced spatio-temporal reasoning required for furniture assembly videos.

Reference graph

Works this paper leans on

85 extracted references · 47 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 17

    Pith/arXiv arXiv 2024

  2. [2]

    Vicas: A dataset for combining holistic and pixel-level video un- derstanding using captions with grounded segmentation

    Ali Athar, Xueqing Deng, and Liang-Chieh Chen. Vicas: A dataset for combining holistic and pixel-level video un- derstanding using captions with grounded segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19023–19035, 2025. 4

    2025

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 4

    Pith/arXiv arXiv 2025

  4. [4]

    PySceneDetect: Video Scene Cut Detection

    Brandon Castellano and contributors. PySceneDetect: Video Scene Cut Detection. https://www.scenedetect. com/, 2025. Version 0.6.6 (released March 9, 2025). 5, 7

    2025

  5. [5]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural In- formation Processing Systems, 37:19472–19495, 2024. 15, 17

    2024

  6. [6]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 9

    2024

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 4

    Pith/arXiv arXiv 2024

  8. [8]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 4

    Pith/arXiv arXiv 2024

  9. [9]

    Per- ceptionlm: Open-access data and models for detailed visual understanding

    Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Tri- antafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Per- ceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180, 2025. 4

    Pith/arXiv arXiv 2025

  10. [10]

    Unifying Specialized Visual Encoders for Video Language Models

    Jihoon Chung, Tyler Zhu, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, and Olga Russakovsky. Unifying specialized visual encoders for video language models. arXiv preprint arXiv:2501.01426, 2025. 4

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Videorefer benchmark evaluation for general mllms

    DAMO-NLP-SG. Videorefer benchmark evaluation for general mllms. https : / / github . com / DAMO - 18 NLP - SG / VideoRefer / blob / main / benchmark / evaluation_general_mllms.md , 2024. Accessed: 2025-06-29. 12

    2024

  12. [12]

    Mevis: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), 2023. 9

    work page 2023

  13. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. 10, 11

    work page 2025

  14. [14]

    Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees G. M. Snoek. Actor and action video segmentation from a sentence. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2018. 9

    work page 2018

  15. [15]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xi- aoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 3302–3310, 2025. 17

    work page 2025

  16. [16]

    Omni-rgpt: Unifying image and video region-level understanding via token marks

    Miran Heo, Min-Hung Chen, De-An Huang, Sifei Liu, Sub- hashree Radhakrishnan, Seon Joo Kim, Yu-Chiang Frank Wang, and Ryo Hachiuma. Omni-rgpt: Unifying image and video region-level understanding via token marks. arXiv preprint arXiv:2501.08326, 2025. 4, 17

    Pith/arXiv arXiv 2025

  17. [17]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 4

    Pith/arXiv arXiv 2024

  18. [18]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In CVPR, 2024. 17

    work page 2024

  19. [19]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2758–2766, 2017. 8

    work page 2017

  20. [20]

    Referring to any person, 2025

    Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Qin Liu, and Lei Zhang. Referring to any person, 2025. 4, 5, 8

    work page 2025

  21. [21]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xin- tao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. Advances in Neural Information Processing Systems, 37:48955–48970, 2024. 8, 17

    work page 2024

  22. [22]

    Large-scale Pre-training for Grounded Video Caption Generation

    Evangelos Kazakos, Cordelia Schmid, and Josef Sivic. Large-scale pre-training for grounded video caption gener- ation. arXiv preprint arXiv:2503.10781, 2025. 6

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34: 11846–11858, 2021. 9, 10, 11, 12

    work page 2021

  24. [24]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 4

    Pith/arXiv arXiv 2024

  25. [25]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 4, 17

    Pith/arXiv arXiv 2023

  26. [26]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 4, 17

    work page 2024

  27. [27]

    Temporal reasoning transfer from text to video

    Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. In ICLR 2025. Open- Review.net, 2025. 18

    work page 2025

  28. [28]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. 2024. 4

    work page 2024

  29. [29]

    Describe anything: Detailed localized image and video captioning

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072 , 2025. 17

    Pith/arXiv arXiv 2025

  30. [30]

    Unleashing hour-scale video train- ing for long video-language understanding

    Jingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Xiaodong Yu, Hao Chen, Jiebo Luo, Zicheng Liu, et al. Unleashing hour-scale video train- ing for long video-language understanding. arXiv preprint arXiv:2506.05332, 2025. 18

    arXiv 2025

  31. [31]

    Perceive anything: Recognize, explain, caption, and segment anything in images and videos

    Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. arXiv preprint arXiv:2506.05302, 2025. 17

    Pith/arXiv arXiv 2025

  32. [32]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4, 5, 8

    Pith/arXiv arXiv 2023

  33. [33]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 4, 10, 11, 18

    Pith/arXiv arXiv 2024

  34. [34]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiao- juan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417–435. Springer, 2024. 17

    work page 2024

  35. [35]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 4, 8, 17

    Pith/arXiv arXiv 2023

  36. [36]

    Point and ask: Incorporating pointing into visual question answering

    Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Rus- sakovsky. Point and ask: Incorporating pointing into visual question answering. arXiv preprint arXiv:2011.13681, 2020. 4

    Pith/arXiv arXiv 2011

  37. [37]

    Pg-video-llava: Pixel grounding large video- language models

    Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and 19 Fahad Khan. Pg-video-llava: Pixel grounding large video- language models. arXiv preprint arXiv:2311.13435 , 2023. 6

    Pith/arXiv arXiv 2023

  38. [38]

    Videoglamm: A large multimodal model for pixel-level vi- sual grounding in videos

    Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: A large multimodal model for pixel-level vi- sual grounding in videos. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 19036– 19046, 2025. 4

    work page 2025

  39. [39]

    Momen- tor: Advancing video large language model with fine-grained temporal reasoning

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat- Seng Chua, Yueting Zhuang, and Siliang Tang. Momen- tor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402.11435, 2024. 17

    Pith/arXiv arXiv 2024

  40. [40]

    Artemis: Towards referential understanding in com- plex videos

    Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in com- plex videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 17

    work page 2024

  41. [41]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009–13018, 2024. 17

    work page 2024

  42. [43]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4, 5, 22, 30

    Pith/arXiv arXiv 2024

  43. [44]

    Grounded sam 2: Ground and track anything in videos with grounding dino, florence-2 and sam 2

    Tianhe Ren, Shuo Shen, et al. Grounded sam 2: Ground and track anything in videos with grounding dino, florence-2 and sam 2. GitHub repository, 2025. https://github. com/IDEA-Research/Grounded-SAM-2 . 4, 5, 13, 14, 24, 25

    work page 2025

  44. [45]

    xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms

    Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, et al. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms. arXiv preprint arXiv:2410.16267, 2024. 4, 8, 11, 12, 29

    Pith/arXiv arXiv 2024

  45. [46]

    Numerologic: Num- ber encoding for enhanced llms’ numerical reasoning

    Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle. Numerologic: Num- ber encoding for enhanced llms’ numerical reasoning. arXiv preprint arXiv:2404.00459, 2024. 31

    Pith/arXiv arXiv 2024

  46. [47]

    Sama: Towards multi-turn referen- tial grounded video chat with large language models

    Ye Sun, Hao Zhang, Henghui Ding, Tiehua Zhang, Xingjun Ma, and Yu-Gang Jiang. Sama: Towards multi-turn referen- tial grounded video chat with large language models. arXiv preprint arXiv:2505.18812, 2025. 4, 17

    arXiv 2025

  47. [48]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 5, 8

    work page 2024

  48. [49]

    Natural language processing with Python and spaCy: A practical introduction

    Yuli Vasiliev. Natural language processing with Python and spaCy: A practical introduction. No Starch Press, 2020. 5

    work page 2020

  49. [50]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024. 8, 17, 31

    Pith/arXiv arXiv 2024

  50. [51]

    Elysium: Exploring object-level perception in videos via mllm

    Han Wang, Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm. In European Conference on Computer Vision , pages 166–185. Springer, 2024. 17

    work page 2024

  51. [52]

    Tarsier: Recipes for training and evaluating large video description models, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models, 2024. 4, 5, 8

    work page 2024

  52. [53]

    Language as queries for referring video object segmen- tation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmen- tation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2022. Also explores Ref-Youtube-VOS, Ref-DA VIS17, A2D-Sentences, JHMDB-Sentences. 9

    work page 2022

  53. [54]

    Longvitu: In- struction tuning for long-form video understanding

    Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan, Yuxuan Wang, Haozhe Zhao, Qing Li, and Yizhou Wang. Longvitu: In- struction tuning for long-form video understanding. arXiv preprint arXiv:2501.05037, 2025. 18

    Pith/arXiv arXiv 2025

  54. [55]

    Number it: Temporal grounding videos like flipping manga

    Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 13754–13765, 2025. 8, 12, 17

    work page 2025

  55. [56]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 1, 8

    work page 2021

  56. [57]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 8

    work page 2017

  57. [58]

    Pixel- aligned language model

    Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Arnab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixel- aligned language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13030–13039, 2024. 17

    work page 2024

  58. [59]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 4

    Pith/arXiv arXiv 2024

  59. [60]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 4

    Pith/arXiv arXiv 2024

  60. [61]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 8 20

    arXiv 2024

  61. [62]

    List items one by one: A new data source and learning paradigm for multimodal llms

    An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. List items one by one: A new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404.16375, 2024. 17

    Pith/arXiv arXiv 2024

  62. [63]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 8, 12

    Pith/arXiv arXiv 2023

  63. [64]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 17

    Pith/arXiv arXiv 2023

  64. [65]

    Merlin: Empowering multimodal llms with foresight minds

    En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xi- angyu Zhang, et al. Merlin: Empowering multimodal llms with foresight minds. In European Conference on Computer Vision, pages 425–443. Springer, 2024. 17

    work page 2024

  65. [66]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 8

    work page 2019

  66. [67]

    Osprey: Pixel un- derstanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel un- derstanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202–28211, 2024. 17

    work page 2024

  67. [68]

    Videorefer suite: Advancing spatial- temporal object understanding with video llm

    Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Bo- qiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial- temporal object understanding with video llm. In Proceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 18970–18980, 2025. 6, 8, 9, 11, 12, 15, 17, 18

    work page 2025

  68. [69]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 5, 7

    work page 2023

  69. [70]

    Videollama 3: Frontier multi- modal foundation models for image and video understand- ing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multi- modal foundation models for image and video understand- ing. arXiv preprint arXiv:2501.13106, 2025. 4

    Pith/arXiv arXiv 2025

  70. [71]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19–35. Springer, 2024. 17

    work page 2024

  71. [72]

    Ferret- v2: An improved baseline for referring and grounding with large language models

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret- v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 , 2024

    Pith/arXiv arXiv 2024

  72. [73]

    Gpt4roi: Instruction tuning large language model on region- of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region- of-interest. In European Conference on Computer Vision , pages 52–70. Springer, 2025. 17

    work page 2025

  73. [74]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 15, 17

    work page 2024

  74. [75]

    active” scene entities can you iden- tify from the video? An entity refers to an object, and “active

    Henghao Zhao, Ge-Peng Ji, Rui Yan, Huan Xiong, and Zechao Li. Videoexpert: Augmented llm for temporal-sensitive video understanding. arXiv preprint arXiv:2504.07519, 2025. 17 21 A. Appendix / Supplemental Material A.1. More Qualitative Results A.1.2 Strefer-Synthesized Data A.1.2 Strefer-Trained Model A.2. More Strefer Details A.3. Model Details A.3.1 Arc...

    Pith/arXiv arXiv 2025

  75. [76]

    Frames are extracted only from the seg- ment of the video

    Ask the model to describe the behavior of entities that are present in a segment of the video. Frames are extracted only from the seg- ment of the video. Template OE Question: <video>What is happening to the woman? Answer: The woman is engaged in a dance with the man, involving spins and turns. She is lifted off the ground by the man during the dance. Que...

    work page

  76. [77]

    Sorry, I’m not sure

    Ask the model to describe the behavior of entities that are not present in a segment of the video; the model should re- spond with uncertainty (e.g., “Sorry, I’m not sure”). Frames are extracted only from the seg- ment of the video. Template OE Question: <video>What is currently hap- pening to the person in a green hoodie? Answer: The person in a green ho...

    work page

  77. [78]

    Frames are extracted only from the seg- ment of the video

    Ask a yes/no question about the presence of an entity in a segment of the video; if present, the model should de- scribe its behavior; if absent, the model should respond with uncertainty. Frames are extracted only from the seg- ment of the video. Template OE Question: <video>Were you able to see a woman in a black jacket? Answer: Yes. The woman walks tow...

    work page

  78. [79]

    Yes” or “No

    Ask a yes/no question about the presence of an entity in a segment of the video; the model should respond with a concise “Yes” or “No” only. Frames are extracted only from the seg- ment of the video. Template OE Question:<video> Is there a woman in a black jacket? Answer only “Yes” or “No”. Answer: Yes. N/A

    work page

  79. [80]

    Frames are extracted from the full video

    Ask the model to iden- tify the correct temporal or- der in which entities first ap- pear in the video from multi- ple choices. Frames are extracted from the full video. Template MCQ Question:<video> Which order shows their first appearance in the video? (A) child interacting with the plant bed, child holding a bag and a toy, child walking across the lawn...

    work page

  80. [81]

    Frames are extracted from the full video

    Ask the model to describe the behavior of entities that may or may not be present in a specific time range of the video; the question refers to a time range. Frames are extracted from the full video. Template OE Question:<video> Could you explain what the girl in the yellow coat is doing between 00:00:05 and 00:00:12.210? Answer: The girl in the yellow co...

    work page

Showing first 80 references.