Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Four-step object reasoning lifts video AI answers by 17.4 points.

desk verdict A useful dataset contribution wrapped in an unsupported method claim; the headline gains come from oracle answers, not from models doing CoT. read the letter →

arxiv 2507.13609 v1 pith:22YEYZH4 submitted 2025-07-18 cs.CV cs.CL

Yanan Wang , Julio Vizcarra , Zhi Li , Hao Niu , Mori Kurokawa This is my paper
classification cs.CVcs.CL
keywords chain-of-thoughtreasoningvideoquestionansweringlargelanguagemodelsinstructiontuningobject-centricframelocalizationentitytrackingspatiotemporalrelationextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoTasks claims that the reason video language models give shallow answers is that they never reason about specific objects across frames, and that this can be repaired by decomposing each high-level video question into four object-level subtasks: frame localization, entity tracking, spatial relation extraction, and temporal relation extraction. The paper builds datasets for NeXT-QA and STAR in which each original question carries these four subtask answers, then feeds the original question plus those answers to several VideoLLMs at inference. On NeXT-QA, every tested model improves, and the lightest model, Qwen2.5-VL-3B, rises from 27.8 to 45.2 average GPT-4 score, with gains of 14.6, 10.9, and 48.1 points in causal, temporal, and descriptive subcategories. On STAR, prompting Qwen2.5-VL-3B with ground-truth CoTasks raises accuracy from 31.1 to 65.4, outperforming even fine-tuning on the original training set. The paper's case is that structured entity-level reasoning context is a cheap, retraining-free route to substantially better compositional video understanding, especially for small models.

What carries the argument

The load-bearing mechanism is the CoTask decomposition pipeline: every high-level question is re-expressed as four dependent subtasks, where each answer feeds the next. CoTask 1 grounds the entities and timestamps; CoTask 2 produces bounding boxes for the grounded entities in those frames; CoTask 3 derives spatial relations within frames; CoTask 4 derives temporal relations across frames. For NeXT-QA, the paper merges VidOR's object annotations by matching video IDs and uses Qwen2.5-VL-72B to generate CoTask 1 answers because the original questions are not frame-specific; for STAR, the grounded objects and frames are taken directly from the dataset. These four answers, formatted as JSON, are then appended to the original question at inference time, which is what the paper means by CoT-style instruction tuning context.

What would settle it

Take a random sample of NeXT-QA questions, have human annotators judge whether the 72B-generated CoTask 1 entities and timestamps actually match the videos, and then replace any incorrect CoTask answers with random or wrong ones; if response scores stay high, the reported gains come from added context text rather than correct object-level grounding, and if they drop sharply, the claim depends on the teacher's hidden accuracy.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that injecting four grounding subtasks—frame localization, entity tracking, spatial relation extraction, and temporal relation extraction—as structured context before answering a video question significantly improves the output quality of VideoLLMs, and the paper reports consistent gains across Qwen2.5-VL-3B/7B/72B and LLaVA-Video-7B. The headline result is Qwen2.5-VL-3B gaining +17.4 average GPT-4 evaluation points (27.8 to 45.2) on NeXT-QA, including +48.1 on descriptive questions, and on STAR the same model jumping from 31.1% to 65.4% accuracy with CoTask prompting. The paper is explicit that the evaluation prompts models with the ground-truth answers to the four subtasks, so the numbers are an upper bound on what the structured context can contribute; no architectural change or retraining is involved. The study also reports that off-the-shelf 7B models solve the individual subtasks poorly (frame localization 17.7%, tracking 1.1%), indicating that the subtask structure reveals a real gap in current models.

Load-bearing premise

The entire result depends on the 72B model's automatic selection of which entities and timestamps each NeXT-QA question refers to being correct, and that selection is never validated—even though the same task is solved at only 17.7% by the smaller models the framework is meant to improve.

Editorial extensions

If this is right

  • If the reported gains hold, lightweight VideoLLMs can approach much larger models' reasoning quality simply by being given object-level context: Qwen2.5-VL-3B with CoTasks reaches 45.2 average while Qwen2.5-VL-72B without CoTasks scores 49.7 on NeXT-QA.
  • Because the evaluation uses ground-truth subtask answers, the real-world benefit depends on generating those answers accurately—a step the ablation shows current small models mostly fail at (tracking: 1.1%), so the practical deployment path is to train or distill a reliable subtask predictor.
  • The four-task decomposition can be reused as an instruction-tuning curriculum, not just inference context; the paper explicitly leaves fine-tuning on CoTasks as future work.
  • On STAR, ground-truth CoTask prompting beats fine-tuning on the original QA pairs by 13.8 points, suggesting that structured intermediate supervision carries more signal than high-level question-answer pairs alone for this model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: run the same protocol with CoTask answers predicted by the model itself; the paper's Table 8 predicts a steep drop, so the practical gain of the framework depends on closing the gap between teacher-generated and self-generated grounding.
  • The unvalidated 72B-generated grounding for NeXT-QA is a hidden confound: if those entity and frame choices are wrong, the downstream CoTasks and final answer inherit the error, and the reported upper bound may owe more to the teacher's hidden ground-truth knowledge than to the CoTask structure itself.
  • One could also test whether the same four subtasks transfer to open-domain video benchmarks without object-level annotations by generating them with a video-grounded LLM and human-checking a sample.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CoTasks, a framework that decomposes video question answering into four intermediate subtasks—frame localization, object tracking, spatial relation extraction, and temporal relation extraction—using object-level annotations from VidOR and STAR. It constructs two datasets (CoTasks-NeXT-QA and CoTasks-STAR) and evaluates three VideoLLMs at inference time by augmenting the input prompt with the ground-truth answers of the four subtasks. The authors report consistent performance gains, most notably +17.4 points for Qwen2.5-VL-3B on NeXT-QA and +34.3 points on STAR.

Significance. If the reported gains had been obtained by models performing the CoTasks themselves, the framework would be valuable for lightweight VideoLLMs. However, the evaluation protocol supplies ground-truth sub-answers directly in the prompt, and the paper's own results show that the evaluated models cannot perform these subtasks (Table 8). The central claim of enabling chain-of-thought reasoning is therefore untested, and the comparison against fine-tuning (Table 7) is not apples-to-apples. The constructed datasets may still be a useful resource for future instruction-tuning research, but the current evidence does not support the paper's conclusions.

major comments (5)
  1. [Sec. 5.3, Table 13] The performance gains reported in Table 6 are obtained by appending ground-truth CoTask answers A1–A4 to the input prompt, not by any model-generated reasoning. In Table 13, the example prompt shows A4 containing the relation 'carry' with tail '4_handbag'; for the question 'what else does the man in yellow carry', A4 already provides the answer, so the model only needs to copy from the context. The +17.4 gain for Qwen2.5-VL-3B therefore reflects oracle information leakage rather than an improvement in compositional reasoning, and the paper's own Table 8 (e.g., 1.1% on CoTask 2) demonstrates that the evaluated models cannot produce these steps themselves.
  2. [Table 7] The STAR comparison in Table 7 is not an apples-to-apples evaluation: the 'PT' condition supplies the model with ground-truth CoTask annotations for the test samples, whereas the 'FT' baseline receives no such oracle. The reported +34.3 point gain over fine-tuning thus conflates the effect of CoTasks with the effect of revealing the answer structure, and cannot support the claim that CoTasks offer 'benefits beyond conventional instruction tuning' (Section 5.3).
  3. [Abstract and Section 6] The claims that CoTasks 'enables models to explicitly perform object-centric spatiotemporal reasoning' and 'substantially improve inference performance' are not supported by the experimental protocol, since no evaluated model ever generates a CoTask step; all intermediate answers come from VidOR/STAR annotations or from Qwen2.5-VL-72B (Section 4.2.2), and Section 5.4 reports that the best model achieves only 17.7% on CoTask 1 and 1.1% on CoTask 2. The paper tests a prompting-with-oracle strategy, not a CoT supervision framework.
  4. [Sec. 4.2.2] The automatic grounding of entities and frames for NeXT-QA is performed by Qwen2.5-VL-72B without any validation of its accuracy, yet the paper's own difficulty analysis (Table 8) suggests that frame localization is very challenging for current VideoLLMs. Since all four CoTasks and the final-answer prompt are built on A1, any error in this step propagates through the entire pipeline; this load-bearing assumption is unverified.
  5. [Title and abstract] The paper is presented as an instruction-tuning framework ('CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks'), but no instruction tuning is performed; the experiments only use the CoTask answers as input prompts (Section 5.3), and the Limitations section concedes that fine-tuning is left to future work. The reported results therefore do not substantiate the claim that CoTasks are effective as a training framework, only as an oracle-annotated prompting scheme.
minor comments (5)
  1. [Table 13] The example in Table 13 is internally inconsistent: A1 and A2 list the entity '3_handbag', while A4 uses '4_handbag', and the expected answer 'book' does not correspond to the carried object in A4 ('handbag'). This also illustrates how directly A4 reveals the answer for such questions.
  2. [Sec. 5.4] The text in Section 5.4 states that CoTasks 1–2 yield +6.9 on DO and +7.3 on DL, but Table 5 shows +6.2 (72.3→78.5) and +9.1 (63.0→72.1); the numbers should be corrected.
  3. [Sec. 4.2.1] The statement that 64-frame uniform sampling is 'empirically validated to be optimal' (Section 4.2.1) is not supported by any comparison experiment in the paper; Table 5 and Table 6 do not vary the number of frames.
  4. [Throughout] The paper uses inconsistent spacing in the model name 'LLaVA-video' (e.g., 'LLaV A' in the abstract and introduction), which should be standardized.
  5. [Appendix A.5] The appendix states that CoTasks will be open-sourced after review, but no data or code link is provided; given the centrality of the constructed dataset, a pointer or release plan would help reproducibility.

Circularity Check

2 steps flagged · score 8.0 of 10

Oracle-only evaluation: reported gains come from feeding ground-truth CoTask answers to the model, so the claim that CoTasks enable chain-of-thought reasoning is not demonstrated.

  1. fitted input called prediction [Section 1 and Table 13; results reported in Section 5.3, Table 6]
    "To assess the upper bound of performance gains, we prompt models with the original question augmented by the ground-truth answers of the CoTasks subtasks."

    The claimed benefit—that CoTasks enable models to explicitly perform object-centric chain-of-thought reasoning—is evaluated by injecting ground-truth CoTask answers A1–A4 into the prompt before the model answers Q0. The evaluated models never generate any CoTask step, so the +17.4 GPT-4 score gain for Qwen2.5-VL-3B measures privileged oracle information, not a model capability. In many cases the final answer is directly contained in the supplied evidence: for a question like 'what else does the man in yellow carry…', A4 states the carry relation and the tail entity. Table 8 confirms the models cannot solve the CoTasks themselves (LLaVA-video-7B achieves only 17.7%, 1.1%, 24.2%, and 10.8% on CoTasks 1–4), so the improved final answers cannot be attributed to model reasoning.

  2. fitted input called prediction [Section 5.3, Table 7 (STAR comparison)]
    "Notably, this performance also surpasses that of the model fine-tuned on the STAR training set by +13.8%, demonstrating that structured grounding and relational context provided through CoTasks offer benefits beyond conventional instruction tuning."

    The STAR comparison is not apples-to-apples: the prompted condition receives ground-truth CoTask annotations (object tracks, spatial relations, action relations) as part of the input, while the fine-tuned baseline receives only the original question and must produce an answer from the video. The +34.3 point prompted-over-baseline gain therefore reflects the injection of oracle answers into the prompt rather than a capability learned or exercised by the model. This comparison cannot support the claim that CoTasks offer benefits 'beyond conventional instruction tuning,' since the fine-tuned baseline is never given the same privileged intermediate answers.

full rationale

The CoTask construction pipeline itself is not circular: decomposing questions into frame localization, tracking, and relation extraction is a legitimate dataset-construction activity, and the paper does not rely on a self-citation chain or imported uniqueness theorem. The circularity is in the evaluation of the central claim. The paper explicitly states that it prompts models with the ground-truth answers of the CoTasks subtasks, and Table 13 instructs the model to use A1–A4 when answering Q0. Because no evaluated model generates any CoTask step, the reported performance gains reduce by construction to the supplied oracle annotations. The paper's own Table 8 and Limitations section acknowledge this setup, but the abstract and conclusion nonetheless present the improvements as evidence that CoTasks 'enables models to explicitly perform object-centric spatiotemporal reasoning' and 'substantially improve inference performance.' That inference is not supported: the experiment measures the value of privileged input, not a reasoning capability. Score is 8 rather than 10 because the final answers are not literally identical to the CoTask inputs in every case, but the central claim is forced by the oracle-conditioned evaluation design.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on external annotation quality, an unvalidated automatic grounding step, and a single LLM-based evaluator. No new physical entities are introduced. The four CoTasks are a task decomposition, not an invented entity. These assumptions are load-bearing because the oracle-prompts experiments inherit every error in the annotation and grounding pipeline.

free parameters (2)
  • 64-frame uniform sampling = 64
    The authors select 64 frames per video, stated in Section 4.2.1 as empirically validated optimal for LLaVA-Video; this sampling rate is a design choice that affects all CoTask construction and evaluation results.
  • timestamp count limit = 1-16
    Prompt templates constrain CoTask 1 output to at most 16 timestamps; this constraint shapes the grounding answers and is chosen by the authors, not derived from data.
assumptions (4)
  • domain assumption VidOR object annotations align with NeXT-QA videos and are accurate
    Section 4.2.1 merges VidOR annotations into NeXT-QA by matching video_ids; if the alignment or annotation accuracy fails, every CoTask is wrong.
  • domain assumption Qwen2.5-VL-72B correctly grounds entities and frames for NeXT-QA questions
    Section 4.2.2 relies on this model to generate CoTask 1 answers for NeXT-QA; no accuracy validation is reported, and Table 8 shows smaller models perform poorly at this task.
  • domain assumption GPT-4 evaluation scores are a valid measure of answer quality
    All headline numbers come from a single GPT-4-based rubric (Section 5.1); the paper does not analyze evaluator variance or correlation with human judgment.
  • ad hoc to paper The four CoTasks capture the reasoning needed for the original questions
    The selection of frame localization, tracking, spatial relations, and temporal relations is justified in the appendix by an appeal to bridging object-level and high-level reasoning, not by an independent analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks." pith.science (2026). https://pith.science/paper/22YEYZH4

@misc{pith2026250713609,
  author       = {Pith},
  title        = {Pith review of: CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22YEYZH4}},
  note         = {Machine review of arXiv:2507.13609}
}
read the original abstract

Despite recent progress in video large language models (VideoLLMs), a key open challenge remains: how to equip models with chain-of-thought (CoT) reasoning abilities grounded in fine-grained object-level video understanding. Existing instruction-tuned models, such as the Qwen and LLaVA series, are trained on high-level video-text pairs, often lacking structured annotations necessary for compositional, step-by-step reasoning. We propose CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks, a new framework that decomposes complex video questions of existing datasets (e.g., NeXT-QA, STAR) into four entity-level foundational tasks: frame localization, entity tracking, spatial and temporal relation extraction. By embedding these intermediate CoT-style reasoning steps into the input, CoTasks enables models to explicitly perform object-centric spatiotemporal reasoning. Experiments on the NeXT-QA benchmark show that CoTasks significantly enhance inference performance: LLaVA-video-7B improves by +3.3 points in average GPT-4 evaluation score, and Qwen2.5-VL-3B gains +17.4, with large boosts in causal (+14.6), temporal (+10.9), and descriptive (+48.1) subcategories. These results demonstrate the effectiveness of CoTasks as a structured CoT-style supervision framework for improving compositional video reasoning.

Figures

Figures reproduced from arXiv: 2507.13609 by the authors.

Figure 1
Figure 1. Comparison of LLaVA-video-7B with and without CoTasks. Given a complex video question, di￾rect inference yields a shallow answer (“Sit”), whereas CoTasks guides the model through entity-aware rea￾soning, resulting in a more grounded and descriptive response. language queries. Despite recent advances, a key open challenge remains: how to equip these mod￾els with structured reasoning abilities that reflect the step-by… view at source ↗
Figure 2
Figure 2. The figure illustrates the process of decomposing a VideoQA task ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of question types in the CoTasks [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative case study on CoTasks-NeXT-QA (left) and CoTasks-STAR (right) comparing model inference [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    OmniJigsaw is a self-supervised proxy task that reconstructs shuffled audio-visual clips via joint integration, sample-level selection, and clip-level masking strategies, yielding gains on 15 video, audio, and reasoni...

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [3]

    Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang

    Accessed: 2025-02-04. Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang

    work page 2025

  2. [4]

    Efficiently verifiable quantum advantage on near-term analog quantum simulators

    Gemini: Google’s multimodal ai models. https://deepmind. google/technologies/gemini. Haotian Liu, Xiang Lin, Chunyuan Li, Shiqi Chen, Kevin Lin, Yuheng Xu, Zihan Wang, Jianwei Zhang, and Chunyuan Zhang. 2024a. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2403.08195. Haotian Liu, Chunyuan Zhang, Yuheng Xu, Zihan Wang, Jian...

    work page Pith review arXiv

  3. [5]

    arXiv preprint arXiv:2304.08485

    Llava: Large language and vision assistant. arXiv preprint arXiv:2304.08485. Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chin- tala, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. 2024b. Dynamem: Online dynamic spatio- semantic memory for open world mobile manipula- tion. arXiv preprint arXiv:2411.04999. Peiqi Liu, Yaswanth Orru, Chris Paxton, Nur Muham...

    arXiv

  4. [8]

    arXiv preprint arXiv:2303.15343

    Siglip: Scaling up image-language pretraining with contrastive mixture of experts. arXiv preprint arXiv:2303.15343. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin

    arXiv

  5. [9]

    arXiv preprint arXiv:2409.12191

    Qwen2- vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie

    arXiv

  6. [10]

    arXiv preprint arXiv:2503.12559

    Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503.12559. Jason Wei et al

    arXiv

  7. [12]

    arXiv preprint arXiv:2309.15402

    Video-of-thought: A dy- namic scene graph approach for video reasoning. arXiv preprint arXiv:2309.15402. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li

    arXiv

  8. [2019]

    arXiv preprint arXiv:1907.06987

    A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987. Zhenfang Chen et al. 2024a. Enhancing video-llm reasoning via agent-of-thoughts distillation. arXiv preprint arXiv:2412.01694. Zhenfang Chen et al. 2024b. Understanding long videos with multimodal language models. arXiv preprint arXiv:2403.16998. Google DeepMind

    arXiv 1907

Show all 12 references
  1. [2022]

    arXiv preprint arXiv:2201.11903

    Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenen- baum, and Chuang Gan

    arXiv

  2. [2023]

    arXiv preprint arXiv:2308.12966

    Qwen-vl: A versatile vision-language model for understanding, localiza- tion, text reading, and beyond. arXiv preprint arXiv:2308.12966. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen- bin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, Humen Zhong, Yuanzhi ...

    arXiv

  3. [2024]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    Video-chatgpt: Towards detailed video understanding via large vi- sion and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikae...

    2024

  4. [2025]

    arXiv preprint arXiv:2502.13923

    Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Joao Carreira, Eric Noland, Chloe Hillier, and An- drew Zisserman

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.