{"id":"71449e5f-a40d-4287-a54d-44723949c977","arxiv_id":"2412.15509","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning LLaVA on prior TRECVid video-text pairs improves scores on the VTT24 captioning test set for BLEU, METEOR, CIDEr, and CIDEr-D, while SPICE and STS results are mixed.","lead":"This paper describes a system that fine-tunes the LLaVA vision-language model on previous TRECVid video captioning data to describe new videos. The fine-tuned model beats two off-the-shelf models on most captioning quality metrics, but not on all of them.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion's 'consistently outperforms' is contradicted by Table 1: LV-V wins SPICE, STS1, and STS2, and no significance tests are provided.","rationale":"Reviewing Table 1, the 'consistently outperforms' claim fails on visual inspection: the video-specific model LV-V posts higher SPICE and STS1/STS2 scores than LV-FT. This is not a question of transfer or external validity; it is an internal discrepancy between the conclusion and the reported results. The reader's selected weakest assumption (transfer from VTT16-23 to VTT24) is a real threat to generalization, but the more immediate load-bearing issue is the unsupported universal claim. Without per-query error bars, even the favorable metrics cannot be distinguished from noise, and the 57% CIDEr improvement is unpaired. Therefore the central claim as written should be tempered. The paper still provides a plausible system description; a conditional accept requiring revised conclusions and released per-query scores is appropriate, matching the reader's verdict. I do not see grounds to reject the paper outright, but the conclusion must be revised to match the evidence, and the per-query data should be released to verify which differences are statistically reliable. The reader's transfer concern remains valid and would be a useful addition to a revision, but it is not the most immediate threat to the central claim as stated.","tokens_in":5826,"tokens_out":6269,"duration_ms":52725,"concrete_test":"Ask the authors to release per-query scores for the 300 VTT24 queries for runs 1, 3, and 4, and compute a paired Wilcoxon signed-rank test (or bootstrap 95% CI) for each of the 10 metrics. If LV-V is significantly better than LV-FT on SPICE, STS1, or STS2, or if LV-FT's advantages over LV on BLEU/METEOR/CIDEr/CIDEr-D are not significant, the conclusion must be revised to acknowledge exceptions and statistical uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 claims LV-FT 'consistently outperforms the vanilla and video-specific models, achieving higher scores in both syntactic and semantic metrics.' Table 1 (main task) shows that LV-V (LLaVA-NeXT-Video, the 'video-specific' model) exceeds LV-FT on SPICE (0.156 vs 0.149), STS1 (0.459 vs 0.448), and STS2 (0.447 vs 0.446). SPICE and STS are semantic metrics, so the claim is internally contradicted by the paper's own numbers. No confidence intervals, error bars, or paired significance tests are reported; the only per-query statistic (57% of 300 queries improve in CIDEr) is unpaired and restricted to one metric. Consequently, the paper's headline conclusion is stronger than the evidence supports. The central finding might survive if reinterpreted as 'fine-tuning LLaVA improves over vanilla LLaVA on most metrics,' but as written the claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the PolySmart team's submission to the TRECVid 2024 Video-to-Text (VTT) task. Three systems are compared: vanilla LLaVA-v1.5-7b using a single middle frame (LV), LLaVA-NeXT-Video-7B-DPO using the full video (LV-V), and LLaVA-v1.5-7b fine-tuned on VTT16-23 frame-text pairs (LV-FT). Table 1 gives official BLEU, METEOR, CIDEr, CIDEr-D, SPICE, and STS scores on the main and robustness tasks. The authors argue that fine-tuning substantially improves both syntactic and semantic metrics and that it aligns the distribution of generated text with the target domain, as illustrated by a t-SNE visualization.","tokens_in":5990,"tokens_out":4218,"duration_ms":35168,"significance":"If the central claim were fully supported, the paper would provide a useful system-level demonstration that task-specific fine-tuning of a vision-language model is needed for TRECVid VTT captioning. The strengths of the paper are that it uses official TRECVid evaluation metrics, standard publicly available models, and a large fine-tuning corpus, and it reports direct measurements rather than fitted parameters. However, the headline conclusion is overstated: on the paper's own Table 1, LV-V beats LV-FT on SPICE, STS1, and STS2 in the main task, so the claim of 'consistently outperforms' is contradicted by the presented numbers. No variance or significance assessment accompanies the scores. The paper is best viewed as a short system description; its scientific contribution is modest but potentially useful as a benchmark report.","major_comments":[{"comment":"The conclusion states that 'The fine-tuned LLaVA model (LV-FT) consistently outperforms the vanilla and video-specific models, achieving higher scores in both syntactic and semantic metrics.' This is contradicted by Table 1 in the main task: LV-V exceeds LV-FT on SPICE (0.156 vs. 0.149), STS1 (0.459 vs. 0.448), and STS2 (0.447 vs. 0.446). In the robustness task, LV-V also wins on SPICE (0.158 vs. 0.147) and STS1 (0.456 vs. 0.444). The claim should be restricted to the metrics where LV-FT actually wins, such as BLEU, METEOR, CIDEr, and CIDEr-D, or should be explicitly qualified as 'most' rather than 'both syntactic and semantic' metrics.","section":"Section 4 / Abstract"},{"comment":"No confidence intervals, error bars, or significance tests are reported for any of the scores. The only per-query statistic, that 57% of 300 queries improve in CIDEr, is an unpaired count and does not establish significance for the rankings. Since several differences are small (e.g., STS2 0.446 vs. 0.447 in the main task), the evidence does not support the word 'consistently' without at least a paired bootstrap or a Wilcoxon-type test over the 300 queries.","section":"Table 1 and Section 3.2"},{"comment":"The comparison between LV-FT and LV-V is confounded by architecture and input representation: LV-FT is LLaVA-v1.5-7b with a middle-frame input, whereas LV-V is LLaVA-NeXT-Video-7B-DPO with full-video input and was not fine-tuned. Therefore, differences between these two runs cannot be attributed solely to fine-tuning. The paper should either restrict the 'effect of fine-tuning' discussion to the LV versus LV-FT comparison or explicitly acknowledge this confound when interpreting Table 1.","section":"Section 2.2 and Section 3.2"},{"comment":"The claim that fine-tuning 'shifts the distribution of LV-FT's generated text closer to the target dataset' is supported only by the qualitative t-SNE plots in Figure 3. No quantitative measure, such as a distribution distance, overlap statistic, or perplexity comparison, is provided. This is not as severe as the Table 1 issue, but the discussion should be framed as an observation rather than a demonstrated alignment effect.","section":"Section 3.3.2"}],"minor_comments":[{"comment":"Describing CIDEr and CIDEr-D as 'semantic' metrics is imprecise; these are n-gram overlap metrics. SPICE and STS are more naturally described as semantic measures. Please adjust the terminology.","section":"Section 3.2"},{"comment":"The statement that 'frame-based inputs allow LV to capture detailed content more effectively than LV-V' is not directly supported because LV and LV-V differ in model architecture and training, not only in input modality. This should be softened.","section":"Section 3.1.2"},{"comment":"The 'Rob.' task is abbreviated as 'Robustness' in the caption, but its construction is never described in the text. Please add a sentence explaining what the robustness task consists of.","section":"Table 1"},{"comment":"Reference [1] is cited for TRECVid 2024 but lists the title 'TRECVID 2023 - A Series of Evaluation Tracks in Video Understanding.' The citation should be updated to the TRECVid 2024 overview paper.","section":"References"},{"comment":"The number '699683' frame-text pairs has no thousands separator and appears as 'resulting in699683' in the text. Also, please provide the number of unique videos and any filtering or de-duplication steps used to construct the fine-tuning set.","section":"Section 2.2"},{"comment":"The notation 'X_a,<i' and 'X_q,<i' is not defined. Please clarify that these denote the token sequences preceding position i.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short system report. The main issue is that the conclusion overstates the results, and the missing statistical support is a real weakness for a journal venue. If the authors revise the claims to match Table 1 and add appropriate statistical caveats, the paper could be acceptable as a system description. The comparison with LV-V should be presented as a comparison of systems rather than as a clean ablation of fine-tuning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core thing to know: this is a short, honest TRECVid VTT system description. The new content is the measured result that fine-tuning LLaVA-v1.5-7b on 699,683 frame-text pairs from VTT16-23 improves BLEU, METEOR, CIDEr, and CIDEr-D on VTT24 over vanilla LLaVA. The gains are large in relative terms (e.g., CIDEr-D 0.427 vs 0.323) and the case studies are plausible. It also compares against LLaVA-NeXT-Video, which does better on SPICE and two STS scores — an interesting wrinkle the authors half-acknowledge in Section 3.1.1.\n\nWhat is good: the table is clear, the fine-tuning data construction is stated (frame every 5 frames), and the per-query CIDEr improvement stat (57% of 300) is a small but concrete extra check. The t-SNE plot is qualitative but reasonable supporting evidence. For a competition system paper, this is about the right level of detail.\n\nSoft spots, in order: (1) The conclusion's 'consistently outperforms... in both syntactic and semantic metrics' is not supported by the paper's own Table 1 — LV-V wins SPICE and STS1/STS2, which are semantic metrics. The stress-test note is right. This needs rewording to 'improves over vanilla LLaVA on most metrics, and over LLaVA-NeXT-Video on n-gram overlap metrics.' (2) No error bars, confidence intervals, or paired significance tests. Given 300 queries, a paired bootstrap on CIDEr is cheap and would make the main claim solid. (3) Fine-tuning hyperparameters (epochs, LR, batch size) and any prompt variations are not reported, which makes the numbers less reproducible. (4) The claim that fine-tuning 'aligns' distributions rests on a single t-SNE figure; fine as illustration, not as evidence.\n\nVerdict: this is a modest but real result. It won't reorganize the field, but it is the kind of confirmation teams need when deciding whether to fine-tune an open VLM on prior VTT data. The authors should temper the conclusion and add reproducibility details. I'd send it to review as a workshop/system track paper; with those changes the central claim is acceptable. Worth citing for the VTT24 numbers.","headline":"Fine-tuning LLaVA on prior VTT data helps most metrics, but 'consistently outperforms' is contradicted by the paper's own table.","tokens_in":6524,"tokens_out":1724,"would_cite":true,"duration_ms":14251,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning LLaVA on seven years of TRECVid VTT captions yields a video captioner that beats vanilla LLaVA and LLaVA-NeXT-Video on BLEU, METEOR, CIDEr, and CIDEr-D.","keywords":["TRECVid 2024","Video-To-Text","video captioning","vision-language models","fine-tuning","LLaVA","LLaVA-NeXT-Video","domain adaptation"],"falsifier":"Take the 300 VTT24 queries, split them by overlap between their ground-truth captions and VTT16-23 caption n-grams, and compare LV-FT vs. LV on the low-overlap half; if the fine-tuned model's CIDEr advantage disappears or reverses on those out-of-distribution queries, the paper's transfer claim would be contradicted.","tokens_in":5619,"feed_emoji":"🎬","tokens_out":5861,"duration_ms":43435,"temperature":0.7,"pith_summary":"This paper reports the PolySmart system for the TRECVid 2024 Video-To-Text (VTT) task. It tries to establish that fine-tuning a vision-language model on past VTT video–caption pairs turns a generic image captioner into a competitive video captioner: the fine-tuned LLaVA (LV-FT) beats both vanilla LLaVA (LV) and the video-native LLaVA-NeXT-Video (LV-V) on BLEU, METEOR, CIDEr and CIDEr-D. The claim matters because TRECVid VTT captions have a distinctive style—short, 'who-what-where-when' sentences—that off-the-shelf VLMs do not naturally produce. The paper argues that domain-specific tuning bridges this gap, both in scores and in the t-SNE distribution of generated text.","feed_headline":"Fine-tuned LLaVA beats generic VLMs on TRECVid captions","feed_subtitle":"Training on past VTT captions lifts BLEU, METEOR, CIDEr and pulls text style toward the dataset.","key_machinery":"The load-bearing object is LV-FT, a fine-tuned LLaVA-v1.5-7b model. Its mechanism is the two-stage VLM pipeline: a vision encoder and MLP projecting visual embeddings into the LLM's token space, then causal language modeling conditioning the answer on the query 'Please write a description of this video frame (around 20-30 words), focusing on Who, What, Where, and When.' The fine-tuning stage resamples VTT16-23 videos at one frame per five and trains on the frame-text pairs, adapting the model's output style to the grounded, short-format VTT captions. The evaluation machinery is the TRECVid metric suite (BLEU, METEOR, CIDEr, CIDEr-D, SPICE, STS1-5) and the t-SNE comparison of generated vs. dataset text embeddings.","core_discovery":"The central discovery is that fine-tuning LLaVA-v1.5-7b on 699,683 frame–text pairs built by sampling every fifth frame from the VTT16–VTT23 datasets yields a model whose TRECVid 2024 VTT captions score higher than the same model without fine-tuning and higher than LLaVA-NeXT-Video processing full videos. On the main task, LV-FT reaches CIDEr 0.712 and CIDEr-D 0.427 versus 0.637/0.323 for LV and 0.511/0.015 for LV-V. The paper reports that 171 of 300 queries improved in CIDEr after fine-tuning, with a mean gain of 0.356. It also presents t-SNE embeddings showing that LV-FT's generated text occupies a distribution closer to the VTT16-23 training captions than LV's, and qualitative examples in which LV-FT adds detail ('khaki shirt', 'sunny day') and corrects errors ('beach' vs ground).","pith_inferences":["An uncontrolled comparison: the paper correlates style alignment with score gains, but the causal chain (alignment drives the metric improvement) is inferred rather than proven; a matched-length or matched-vocabulary analysis would isolate the mechanism.","If the t-SNE alignment is the real driver, fine-tuning should transfer to VTT24 mostly when VTT24 captions stylistically resemble VTT16-23; a future year with a more diverse caption set could erase the CIDEr advantage.","Because LLaVA-NeXT-Video was not fine-tuned (no official code), the comparison is unfair to the video model: a fine-tuned video-input model might combine the alignment benefit with temporal understanding, a testable next step.","The frame-sampling strategy (every fifth frame) effectively multiplies the dataset by five; the same approach could be applied to newer VLMs with instruction fine-tuning to further push caption quality."],"forward_implications":["For TRECVid-style video captioning, an image-based VLM fine-tuned on past VTT data can outperform a video-native VLM used out of the box, so teams should consider lightweight frame sampling plus fine-tuning before adopting video-input models.","Fine-tuning on the historical VTT16-23 collection is a reproducible recipe: 699,683 frame-text pairs, five-frame sampling, and the official LLaVA fine-tuning script.","The gain is concentrated in n-gram overlap metrics (BLEU, METEOR, CIDEr, CIDEr-D); SPICE and STS scores do not consistently improve, suggesting fine-tuning mostly sharpens wording and style rather than deep semantic content.","The 171-of-300 query-level CIDEr improvement can serve as a per-example diagnostic for where fine-tuning helps or hurts."],"supporting_citations":[{"why":"Defines the TRECVid 2023 track and the VTT evaluation setup.","marker":"[1]"},{"why":"Provides the LLaVA architecture and visual instruction tuning recipe used to build LV and LV-FT.","marker":"[11]"},{"why":"Supplies LLaVA-NeXT-Video, the video-input baseline LV-V.","marker":"[13]"},{"why":"Gives the improved LLaVA-v1.5 baselines that the fine-tuning procedure follows.","marker":"[14]"},{"why":"Describes the V3C collection from which the VTT16-23 frame-text pairs are sampled.","marker":"[19]"},{"why":"Establishes the TRECVid evaluation methodology and the metrics used in Table 1.","marker":"[20]"}],"fun_headline_variants":["Fine-tuning LLaVA on VTT data boosts CIDEr from 0.637 to 0.712","Domain tuning gives LLaVA a big edge over video-native VLMs","After fine-tuning, LLaVA improves 171 out of 300 VTT captions","Fine-tuned LLaVA aligns caption style to VTT data, lifting CIDEr","TRECVid 2024: fine-tuned LLaVA beats LLaVA-NeXT-Video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the VTT16-23 caption distribution is representative enough of VTT24 that style alignment on the old years raises scores on the new year; if the 2024 captions differ systematically in wording, detail, or annotation style, the reported CIDEr gains may vanish outside the training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning LLaVA on VTT data boosts CIDEr from 0.637 to 0.712","Domain tuning gives LLaVA a big edge over video-native VLMs","After fine-tuning, LLaVA improves 171 out of 300 VTT captions","Fine-tuned LLaVA aligns caption style to VTT data, lifting CIDEr","TRECVid 2024: fine-tuned LLaVA beats LLaVA-NeXT-Video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1681,"prompt_tokens":904,"completion_tokens":777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":657}},"tokens_in":520,"tokens_out":777,"duration_ms":5558,"temperature":1.0,"reasoning_tokens":657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:21:02.271086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 300 VTT24 queries, split them by overlap between their ground-truth captions and VTT16-23 caption n-grams, and compare LV-FT vs. LV on the low-overlap half; if the fine-tuned model's CIDEr advantage disappears or reverses on those out-of-distribution queries, the paper's transfer claim would be contradicted.","supporting_citations":[{"cited_title":"Trecvid 2023 - a series of evaluation tracks in video understanding,","cited_arxiv_id":null,"evidence_quote":"Defines the TRECVid 2023 track and the VTT evaluation setup."},{"cited_title":"Visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA architecture and visual instruction tuning recipe used to build LV and LV-FT."},{"cited_title":"Improved baselines with visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Gives the improved LLaVA-v1.5 baselines that the fine-tuning procedure follows."},{"cited_title":"V3c–a research video collection,","cited_arxiv_id":null,"evidence_quote":"Describes the V3C collection from which the VTT16-23 frame-text pairs are sampled."},{"cited_title":"Evaluationcampaignsand trecvid,","cited_arxiv_id":null,"evidence_quote":"Establishes the TRECVid evaluation methodology and the metrics used in Table 1."}],"review_version":1}