REVIEW 4 major objections 6 minor 20 references
PolySmart @ TRECVid 2024 Video Captioning (VTT)
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Fine-tuning LLaVA on seven years of TRECVid VTT captions yields a video captioner that beats vanilla LLaVA and LLaVA-NeXT-Video on BLEU, METEOR, CIDEr, and CIDEr-D.
desk verdict Fine-tuning LLaVA on prior VTT data helps most metrics, but 'consistently outperforms' is contradicted by the paper's own table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is LV-FT, a fine-tuned LLaVA-v1.5-7b model. Its mechanism is the two-stage VLM pipeline: a vision encoder and MLP projecting visual embeddings into the LLM's token space, then causal language modeling conditioning the answer on the query 'Please write a description of this video frame (around 20-30 words), focusing on Who, What, Where, and When.' The fine-tuning stage resamples VTT16-23 videos at one frame per five and trains on the frame-text pairs, adapting the model's output style to the grounded, short-format VTT captions. The evaluation machinery is the TRECVid metric suite (BLEU, METEOR, CIDEr, CIDEr-D, SPICE, STS1-5) and the t-SNE comparison of generated vs. dataset text embeddings.
What would settle it
Take the 300 VTT24 queries, split them by overlap between their ground-truth captions and VTT16-23 caption n-grams, and compare LV-FT vs. LV on the low-overlap half; if the fine-tuned model's CIDEr advantage disappears or reverses on those out-of-distribution queries, the paper's transfer claim would be contradicted.
Extended reading notes
Core claim
The central discovery is that fine-tuning LLaVA-v1.5-7b on 699,683 frame–text pairs built by sampling every fifth frame from the VTT16–VTT23 datasets yields a model whose TRECVid 2024 VTT captions score higher than the same model without fine-tuning and higher than LLaVA-NeXT-Video processing full videos. On the main task, LV-FT reaches CIDEr 0.712 and CIDEr-D 0.427 versus 0.637/0.323 for LV and 0.511/0.015 for LV-V. The paper reports that 171 of 300 queries improved in CIDEr after fine-tuning, with a mean gain of 0.356. It also presents t-SNE embeddings showing that LV-FT's generated text occupies a distribution closer to the VTT16-23 training captions than LV's, and qualitative examples in which LV-FT adds detail ('khaki shirt', 'sunny day') and corrects errors ('beach' vs ground).
Load-bearing premise
The load-bearing premise is that the VTT16-23 caption distribution is representative enough of VTT24 that style alignment on the old years raises scores on the new year; if the 2024 captions differ systematically in wording, detail, or annotation style, the reported CIDEr gains may vanish outside the training distribution.
Editorial extensions
If this is right
- For TRECVid-style video captioning, an image-based VLM fine-tuned on past VTT data can outperform a video-native VLM used out of the box, so teams should consider lightweight frame sampling plus fine-tuning before adopting video-input models.
- Fine-tuning on the historical VTT16-23 collection is a reproducible recipe: 699,683 frame-text pairs, five-frame sampling, and the official LLaVA fine-tuning script.
- The gain is concentrated in n-gram overlap metrics (BLEU, METEOR, CIDEr, CIDEr-D); SPICE and STS scores do not consistently improve, suggesting fine-tuning mostly sharpens wording and style rather than deep semantic content.
- The 171-of-300 query-level CIDEr improvement can serve as a per-example diagnostic for where fine-tuning helps or hurts.
Reading between the lines
- An uncontrolled comparison: the paper correlates style alignment with score gains, but the causal chain (alignment drives the metric improvement) is inferred rather than proven; a matched-length or matched-vocabulary analysis would isolate the mechanism.
- If the t-SNE alignment is the real driver, fine-tuning should transfer to VTT24 mostly when VTT24 captions stylistically resemble VTT16-23; a future year with a more diverse caption set could erase the CIDEr advantage.
- Because LLaVA-NeXT-Video was not fine-tuned (no official code), the comparison is unfair to the video model: a fine-tuned video-input model might combine the alignment benefit with temporal understanding, a testable next step.
- The frame-sampling strategy (every fifth frame) effectively multiplies the dataset by five; the same approach could be applied to newer VLMs with instruction fine-tuning to further push caption quality.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the PolySmart team's submission to the TRECVid 2024 Video-to-Text (VTT) task. Three systems are compared: vanilla LLaVA-v1.5-7b using a single middle frame (LV), LLaVA-NeXT-Video-7B-DPO using the full video (LV-V), and LLaVA-v1.5-7b fine-tuned on VTT16-23 frame-text pairs (LV-FT). Table 1 gives official BLEU, METEOR, CIDEr, CIDEr-D, SPICE, and STS scores on the main and robustness tasks. The authors argue that fine-tuning substantially improves both syntactic and semantic metrics and that it aligns the distribution of generated text with the target domain, as illustrated by a t-SNE visualization.
Significance. If the central claim were fully supported, the paper would provide a useful system-level demonstration that task-specific fine-tuning of a vision-language model is needed for TRECVid VTT captioning. The strengths of the paper are that it uses official TRECVid evaluation metrics, standard publicly available models, and a large fine-tuning corpus, and it reports direct measurements rather than fitted parameters. However, the headline conclusion is overstated: on the paper's own Table 1, LV-V beats LV-FT on SPICE, STS1, and STS2 in the main task, so the claim of 'consistently outperforms' is contradicted by the presented numbers. No variance or significance assessment accompanies the scores. The paper is best viewed as a short system description; its scientific contribution is modest but potentially useful as a benchmark report.
major comments (4)
- [Section 4 / Abstract] The conclusion states that 'The fine-tuned LLaVA model (LV-FT) consistently outperforms the vanilla and video-specific models, achieving higher scores in both syntactic and semantic metrics.' This is contradicted by Table 1 in the main task: LV-V exceeds LV-FT on SPICE (0.156 vs. 0.149), STS1 (0.459 vs. 0.448), and STS2 (0.447 vs. 0.446). In the robustness task, LV-V also wins on SPICE (0.158 vs. 0.147) and STS1 (0.456 vs. 0.444). The claim should be restricted to the metrics where LV-FT actually wins, such as BLEU, METEOR, CIDEr, and CIDEr-D, or should be explicitly qualified as 'most' rather than 'both syntactic and semantic' metrics.
- [Table 1 and Section 3.2] No confidence intervals, error bars, or significance tests are reported for any of the scores. The only per-query statistic, that 57% of 300 queries improve in CIDEr, is an unpaired count and does not establish significance for the rankings. Since several differences are small (e.g., STS2 0.446 vs. 0.447 in the main task), the evidence does not support the word 'consistently' without at least a paired bootstrap or a Wilcoxon-type test over the 300 queries.
- [Section 2.2 and Section 3.2] The comparison between LV-FT and LV-V is confounded by architecture and input representation: LV-FT is LLaVA-v1.5-7b with a middle-frame input, whereas LV-V is LLaVA-NeXT-Video-7B-DPO with full-video input and was not fine-tuned. Therefore, differences between these two runs cannot be attributed solely to fine-tuning. The paper should either restrict the 'effect of fine-tuning' discussion to the LV versus LV-FT comparison or explicitly acknowledge this confound when interpreting Table 1.
- [Section 3.3.2] The claim that fine-tuning 'shifts the distribution of LV-FT's generated text closer to the target dataset' is supported only by the qualitative t-SNE plots in Figure 3. No quantitative measure, such as a distribution distance, overlap statistic, or perplexity comparison, is provided. This is not as severe as the Table 1 issue, but the discussion should be framed as an observation rather than a demonstrated alignment effect.
minor comments (6)
- [Section 3.2] Describing CIDEr and CIDEr-D as 'semantic' metrics is imprecise; these are n-gram overlap metrics. SPICE and STS are more naturally described as semantic measures. Please adjust the terminology.
- [Section 3.1.2] The statement that 'frame-based inputs allow LV to capture detailed content more effectively than LV-V' is not directly supported because LV and LV-V differ in model architecture and training, not only in input modality. This should be softened.
- [Table 1] The 'Rob.' task is abbreviated as 'Robustness' in the caption, but its construction is never described in the text. Please add a sentence explaining what the robustness task consists of.
- [References] Reference [1] is cited for TRECVid 2024 but lists the title 'TRECVID 2023 - A Series of Evaluation Tracks in Video Understanding.' The citation should be updated to the TRECVid 2024 overview paper.
- [Section 2.2] The number '699683' frame-text pairs has no thousands separator and appears as 'resulting in699683' in the text. Also, please provide the number of unique videos and any filtering or de-duplication steps used to construct the fine-tuning set.
- [Equation (1)] The notation 'X_a,<i' and 'X_q,<i' is not defined. Please clarify that these denote the token sequences preceding position i.
Circularity Check
No significant circularity: the paper's central claim is an external TRECVid benchmark comparison, not a derivation from its own inputs.
full rationale
The paper is an empirical TRECVid VTT system description, not a derivation chain. The only load-bearing comparison is Table 1, where LV-FT is evaluated against official TRECVid 2024 ground-truth captions after fine-tuning on VTT16-23 video-text pairs; VTT24 is not used in training, so the reported gains are a held-out external benchmark measurement, not an identity, a fitted-variable rename, or a prediction forced by construction. The t-SNE analysis in Section 3.3.2 is qualitative post-hoc evidence rather than a load-bearing derivation. Several references include overlapping authors (e.g., refs [5]-[9], [12], [13]), but they appear only as background or contextual support for general claims about video understanding and VLMs; none is invoked as a uniqueness theorem, none smuggles in an ansatz, and none substitutes for the external metric table. Therefore self-citation is not circular here. One non-circular correctness issue is noted: the conclusion's statement that 'LV-FT consistently outperforms the vanilla and video-specific models, achieving higher scores in both syntactic and semantic metrics' is internally contradicted by Table 1, where LV-V exceeds LV-FT on SPICE (0.156 vs 0.149), STS1 (0.459 vs 0.448), and STS2 (0.447 vs 0.446), and no significance tests are reported. This is an overstatement or missing-support concern, not a circularity concern. Since the central claim is evaluated against official external metrics and no reported result reduces to its own inputs, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption VTT16-23 captions are representative of VTT24 captions
- standard math The official TRECVid metrics (BLEU, METEOR, CIDEr, CIDEr-D, SPICE, STS) are valid measures of caption quality
- domain assumption Extracting one frame every five frames preserves the video content needed for captioning
Cite this review
Pith. "Pith review of PolySmart @ TRECVid 2024 Video Captioning (VTT)." pith.science (2026). https://pith.science/paper/HQRPYPJ5
@misc{pith2026241215509,
author = {Pith},
title = {Pith review of: PolySmart @ TRECVid 2024 Video Captioning (VTT)},
year = {2026},
howpublished = {\url{https://pith.science/paper/HQRPYPJ5}},
note = {Machine review of arXiv:2412.15509}
}
read the original abstract
In this paper, we present our methods and results for the Video-To-Text (VTT) task at TRECVid 2024, exploring the capabilities of Vision-Language Models (VLMs) like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content. We investigate the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency. Our analysis reveals that fine-tuning substantially improves the model's ability to produce more detailed and domain-aligned text, bridging the gap between generic VLM tasks and the specialized needs of VTT. Experimental results demonstrate that our fine-tuned model outperforms baseline VLMs across various evaluation metrics, underscoring the importance of domain-specific tuning for complex VTT tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Trecvid 2023 - a series of evaluation tracks in video understanding,
G. Awad, K. Curtis, A. A. Butt, J. Fiscus, A. Godil, Y. Lee, A. Delgado, E. Godard, L. Diduch, Y. Graham, , and G. Quénot, “Trecvid 2023 - a series of evaluation tracks in video understanding,” in Proceedings of TRECVID 2023. NIST, USA, 2023
work page 2023
-
[2]
C.-W. Ngo, Z. Pan, X. Wei, X. Wu, H.-K. Tan, and W. Zhao, “Motion driven approaches to shot boundary detection, low-level feature extraction and bbc rushes characterization at TRECVid 2005,” in TRECVID, 2005
work page 2005
-
[3]
Beyond semantic search: What you observe may not be what you think,
C.-W. Ngo, Y.-G. Jiang, X.-Y. Wei, W. Zhao, F. Wang, X. Wu, and H.-K. Tan, “Beyond semantic search: What you observe may not be what you think,” inIEEE Computer Society , 2008
work page 2008
-
[4]
VIREO at TRECvID 2010: Semantic indexing, known-item search, and content-based copy detection,
C.-W. Ngo, S.-A. Zhu, H.-K. Tan, W.-L. Zhao, and X.-Y. Wei, “VIREO at TRECvID 2010: Semantic indexing, known-item search, and content-based copy detection,” inTRECVID, 2010
work page 2010
-
[5]
VIREO@TRECVid 2023: Ad-hoc video search,
J. Wu, Z. Ma, C.-W. Ngo, and S.-H. Zhong, “VIREO@TRECVid 2023: Ad-hoc video search,” inIn NIST TRECVID Workshop, 2023
work page 2023
-
[6]
VIREO@TRECVid 2022: Ad-hoc video search,
J. Wu, Z. Ma, and C.-W. Ngo, “VIREO@TRECVid 2022: Ad-hoc video search,” in In NIST TRECVID Workshop, 2022
work page 2022
-
[7]
VIREO@TRECVid 2021: Ad-hoc video search,
J. Wu, Z. Hou, Z. Ma, and C.-W. Ngo, “VIREO@TRECVid 2021: Ad-hoc video search,” inIn NIST TRECVID Workshop, 2021
work page 2021
-
[8]
J. Wu, C. wah Ngo, and W.-K. Chan, “Improving interpretable embeddings for ad-hoc video search with generative captions and multi-word concept bank,” inProceedings of the ACM on International Conference on Multimedia Retrieval , 2024, pp. 1–10
work page 2024
Show all 20 references
-
[9]
(Un)likelihood training for interpretable embedding,
J. Wu, C.-W. Ngo, W.-K. Chan, and Z. Hou, “(Un)likelihood training for interpretable embedding,” in ACM Transactions on Information Systems , 2023
2023
-
[10]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[11]
Visual instruction tuning,
H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, 2024
2024
-
[12]
Interactive video search with multi-modal llm video captioning,
Y.-T. Cheng, J. Wu, Z. Ma, J. He, X.-Y. Wei, and C.-W. Ngo, “Interactive video search with multi-modal llm video captioning,” in Proceedings of the International Conference on Multimedia Modelling, 2025, pp. 1–8
2025
-
[13]
Video instruction tuning with synthetic data,
Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li, “Video instruction tuning with synthetic data,” arXiv preprint arXiv:2410.02713 , 2024
2024 arXiv
-
[14]
Improved baselines with visual instruction tuning,
H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26296–26306
2024
-
[15]
Minigpt-4: Enhancing vision-language under- standing with advanced large language models,
D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language under- standing with advanced large language models,”arXiv preprint arXiv:2304.10592 , 2023
2023 arXiv
-
[16]
Llama-vid: An image is worth 2 tokens in large language models,
Y. Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” in European Conference on Computer Vision . Springer, 2025, pp. 323–340
2025
-
[17]
Minigpt4- video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens,
K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, D. Zhu, J. Ding, and M. Elhoseiny, “Minigpt4- video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens,” arXiv preprint arXiv:2404.03413 , 2024
2024 arXiv
-
[18]
Llava-next: Improved reasoning, ocr, and world knowledge,
H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” 2024
2024
-
[19]
V3c–a research video collection,
L. Rossetto, H. Schuldt, G. Awad, and A. A. Butt, “V3c–a research video collection,” inInternational Conference on Multimedia Modeling . Springer, 2019, pp. 349–360
2019
-
[20]
Evaluationcampaignsand trecvid,
A. F. Smeaton, P. Over, and W. Kraaij, “Evaluationcampaignsand trecvid,” inMIR ’06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval . New York, NY, USA: ACM Press, 2006, pp. 321–330
2006
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.