{"id":"29b47e21-47f1-47d3-bf92-f093de5e4d6b","arxiv_id":"2506.01725","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GRPO post-training with structured thinking and dual think/caption rewards improves Qwen2-VL-7B video captioning over the base model and SFT on DREAM-1K, VDC, and CAREBENCH.","lead":"This paper trains a video captioning model with GRPO reinforcement learning to first think through who is in a video, what they look like, and what they do, then write the final caption. On three benchmarks, this approach beats both the base model and supervised fine-tuning with only 1.5k training videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The caption reward (Escore) and the DREAM-1K/VDC evaluation both rely on Qwen2.5-72B as judge; without cross-judge or human validation, the reported RL gains may overfit that judge rather than reflect genuine captioning improvement.","rationale":"I reviewed the paper in good faith. The experimental design is internally consistent: the two-stage generation, think scorer (Sec. 3.2.2), caption scorer (Sec. 3.2.3), and GRPO training are clearly described, and Tables 1–3 show consistent gains over the Qwen2-VL-7B baseline and SFT counterparts on three benchmarks. The ablation in Table 3 is informative, especially the contrast between CNscore and Escore, which demonstrates awareness of reward hacking. However, the most load-bearing assumption—that the reward function captures genuine caption quality—is not independently validated. The paper's default caption reward Escore is an LLM-based event-entailment score generated by Qwen2.5-72B, and the evaluation metrics on DREAM-1K and VDC are themselves LLM-based (DREAM-1K event F1 comes from Tarsier, from which the Escore prompt is taken). Consequently, the RL policy is optimized against a proxy that is structurally aligned with the test metric; without a human or cross-judge evaluation, the observed improvements could reflect overfitting to Qwen2.5-72B's preferences rather than better captioning. The decontamination issue (training videos sampled from Tarsier2-Recap-585K overlapping DREAM-1K) is also valid and unaddressed. Both concerns are testable and would not necessarily refute the paper, so I retain the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT. I agree only partially with the reader's weakest_assumption: the reader identifies noisy/biased reward targets generally, while I emphasize the structural alignment between the reward and the evaluation judge, which makes the noise systematic rather than random.","tokens_in":19348,"tokens_out":6612,"duration_ms":64453,"concrete_test":"Re-run the evaluation of VideoCap-R1 and the SFT baseline on DREAM-1K, VDC, and CAREBENCH with an independent judge not used in training: (a) for DREAM-1K event F1, replace the Qwen2.5-72B event-entailment step with GPT-4o or a different LLM; (b) for VDC and CAREBENCH, either swap the scoring model or have human raters score a 100-video stratified subset. Then compare the deltas (VideoCap-R1 vs SFT) under the original and independent judge. If the relative gains shrink or reverse, the central claim is an artifact of judge overfitting; if they persist, the structured-thinking RL recipe is corroborated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GRPO with structured thinking and dual rewards outperforms SFT for video captioning—depends on the reward signal measuring true caption quality rather than the quirks of a particular judge model. The paper's caption score Escore (Eq. 8) asks Qwen2.5-72B to judge whether ground-truth events are entailed by the predicted caption, using 'the same prompt as Tarsier[33]' (Sec. 3.2.3). DREAM-1K's event-F1 metric, introduced in Tarsier, is computed with the same LLM-based event extraction/entailment protocol; VDC's VDCSCORE and CAREBENCH's action/object F1 also rely on model-based similarity. Thus the model is trained to optimize a judge that is essentially the test metric. The paper itself shows that direct LLM scoring (CNscore) suffers reward hacking (Sec. 4.3), and the authors claim Escore is more objective, but no evidence is provided—no human evaluation, no cross-judge check (e.g., GPT-4o or Llama), no correlation between Escore and human ratings. RL can exploit systematic biases (preferred phrasing, verbosity, or overly specific wording) that inflate this judge's scores and the benchmark metrics simultaneously. Without an independent judge, the +4.4/+4.2/+3.1/+6.9 gains, and the claim that GRPO is superior to SFT, are ambiguous: they may reflect overfitting to Qwen2.5-72B rather than better captions. The decontamination concern raised by the reader is secondary but also unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VideoCap-R1, a GRPO-based reinforcement learning post-training method for video captioning. Starting from Qwen2-VL-7B, the model is prompted to first emit structured thinking that identifies subjects, attributes, and actions, and then produce a full caption. Two reward components are introduced: an LLM-free think score (Tscore) computed by embedding-based bipartite matching against structured annotations, and an LLM-assisted caption score (Escore) based on Qwen2.5-72B event-entailment judgments. Training on a curated 1.5k-video subset of Tarsier2-Recap-585K, the paper reports gains over the Qwen2-VL-7B baseline on DREAM-1K, VDC, and CAREBENCH, and shows that its GRPO-based approach outperforms SFT trained on the same data. Ablations examine the contribution of each reward component and compare Escore with a direct LLM scoring variant (CNscore).","tokens_in":19615,"tokens_out":5981,"duration_ms":58177,"significance":"If the empirical claims hold, this is a useful contribution: it is one of the first systematic applications of GRPO to open-ended video captioning, and the central comparison in Table 2—RL versus SFT on identical 1.5k training samples—is an internally consistent and relevant experiment. The paper also provides a careful decomposition of the reward into thinking and caption components, with ablations for each, and shows qualitative examples of finer-grained action description. The main risk is that the reward signal and the evaluation metrics are produced by the same Qwen2.5-72B judge, so the reported gains may partly reflect overfitting to that judge rather than true caption-quality improvement. The lack of multiple seeds and the absence of a decontamination check for the Tarsier-derived training set further weaken the current evidence. With additional validation, the method could be a valuable data-efficient alternative to SFT for video captioning.","major_comments":[{"comment":"The Escore reward is computed by Qwen2.5-72B entailment judgments using the same prompt as Tarsier, and the DREAM-1K event F1 metric is defined by the same Tarsier protocol; VDCSCORE and CAREBENCH action/object F1 also rely on model-based similarity. The RL policy is therefore trained to optimize a reward that is essentially the evaluation metric. The paper does not provide cross-judge validation (e.g., GPT-4o or Llama), human ratings, or a correlation analysis between Escore and human judgments. Without such evidence, the claimed gains of +4.4/+4.2/+3.1/+6.9 and the conclusion that GRPO is superior to SFT remain ambiguous, because both could reflect overfitting to the Qwen2.5-72B judge.","section":"Sec. 3.2.3, Eq. (8); Sec. 4.1"},{"comment":"All reported results come from a single run, with no error bars or multiple seeds. The differences between SFT and RL, such as 32.8 vs. 34.2 DREAM-1K event F1 and 31.5 vs. 34.4 CAREBENCH action F1, are modest and could be within run-to-run variance. To support the central claim of GRPO superiority, the authors should report the mean and variance over at least three independent training runs, or otherwise justify that the observed differences are statistically meaningful.","section":"Table 2; Sec. 4.2.1"},{"comment":"The training set is sampled from Tarsier2-Recap-585K, and the DREAM-1K benchmark originates from the Tarsier/LLM-based evaluation family. The paper does not report any decontamination or overlap analysis between the 1.5k training videos and the DREAM-1K, VDC, or CAREBENCH test sets. If overlap exists, the DREAM-1K improvements could be inflated. The authors should measure exact or near-duplicate video overlap and report results on the non-overlapping subset.","section":"Appendix B; Sec. 4.1"},{"comment":"The thresholds δ1=0.28 and δ2=0.35 in Escore and the weights 0.6/0.4 in Tscore are fixed values without sensitivity analysis. If these hyperparameters were selected using the evaluation benchmarks, the held-out claim is weakened. The authors should provide a sensitivity study over δ1, δ2, and the Tscore weights, or otherwise justify that the results are robust to reasonable variations.","section":"Eq. (6), Eq. (8), Appendix A"}],"minor_comments":[{"comment":"The fourth data row, reporting values 32.5, 46.8, 35.2, 31.6, and 36.5, is missing its model label; from the text it appears to be the CNscore configuration, but the table should make this explicit.","section":"Table 3"},{"comment":"The last column header appears as 'A VG' and should be 'AVG' (average).","section":"Table 2"},{"comment":"The three subfigures lack axis labels and legends, making it difficult to know which curve corresponds to which metric and what the x-axis represents.","section":"Figure 2"},{"comment":"The think scorer is described as LLM-free, which is accurate at scoring time, but the ground-truth structured annotations used for matching are generated by Qwen2.5-72B from source captions (Appendix B). The paper should clarify this distinction to avoid implying that the annotations are human-verified.","section":"Sec. 3.2.2"},{"comment":"The statement that the Escore prompt is 'the same as Tarsier' should cite the exact prompt or include it in the appendix, since the equivalence between the reward prompt and the evaluation protocol is load-bearing for the central claim.","section":"Sec. 3.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a computer vision journal and addresses a timely topic. The main concern is whether the reported gains reflect genuine captioning improvements or overfitting to the Qwen2.5-72B judge; this is fixable with additional experiments. I would encourage the editor to request cross-judge and human evaluation, multiple seeds, and a decontamination analysis before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper gives a credible, data-efficient RL recipe for video captioning, but its headline claim—RL beats SFT—is less clean than it looks because the reward function and the benchmarks are scored by the same judge protocol.\n\nWhat is actually new: structured thinking that decomposes captioning into subject/attribute/action lists before the final description; a think score computed via embedding-based bipartite matching (no LLM needed at training time); and an event-coverage caption score with two thresholds. The ablations are honest: they show direct LLM scoring (CNscore) gets reward-hacked and switch to event coverage. They also show SFT with structured thinking does not help, while GRPO does—that is a genuinely useful observation for people trying R1-style RL on open-ended tasks.\n\nThe soft spots are concentrated where the reader guessed. No error bars or seeds, thresholds fixed with no sensitivity, and no decontamination check between Tarsier2-Recap-585K training data and DREAM-1K, which comes from the same lineage. The bigger problem is judge overlap: Escore (Eq. 8) asks Qwen2.5-72B to judge event entailment using the same prompt as Tarsier, and DREAM-1K's event-F1 uses the same protocol. VDC and CAREBENCH also lean on model-based scoring. So the +4.4 event F1 and +4.2 VDC gains may be the model learning to satisfy Qwen2.5-72B rather than producing better captions by human standards. Without cross-judge or human evaluation, the 'GRPO superiority' claim is under-determined.\n\nThat said, the central direction is sound and the recipe is reproducible in principle. The paper deserves a serious referee, but the referee should ask for multiple runs, a leakage/overlap analysis, threshold sensitivity, and an independent judge or human study. It's not a desk reject; it's a revise-and-resubmit candidate. I'd point a student doing RL for video MLLMs to it, with the judge-overfitting caveat on page one.","headline":"A credible, data-efficient RL recipe for video captioning, but the RL-vs-SFT claim is clouded because the reward and the benchmarks share the same Qwen2.5-72B judge protocol.","tokens_in":20281,"tokens_out":2500,"would_cite":false,"duration_ms":24408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RL post-training lifts video caption accuracy with only 1.5k samples","keywords":["video captioning","reinforcement learning","GRPO","multimodal large language model","structured thinking","reward modeling","action description","chain-of-thought"],"falsifier":"Take 100 held-out videos and have human annotators rate VideoCap-R1 captions against SFT and baseline captions for factual correctness; if human preference does not align with the reported F1/accuracy gains, the rewards are optimizing a proxy. A sharper version: retrain with human-written structured annotations instead of Qwen2.5-72B-generated ones and check whether the DREAM-1K event-F1 gain of +4.4 persists.","tokens_in":19049,"feed_emoji":"🎬","tokens_out":5513,"duration_ms":49167,"temperature":0.7,"pith_summary":"The paper tries to establish that group-relative policy optimization (GRPO) reinforcement learning can improve open-ended video captioning in a 7B multimodal language model, using only 1.5k training videos and without large supervised datasets. The proposed model, VideoCap-R1, is prompted to first produce structured thinking that lists each video subject's attributes and actions, then write a full caption; two rewards score the thinking and the caption. On three benchmarks the authors report consistent gains over the Qwen2-VL-7B baseline (DREAM-1K +4.4 event F1, VDC +4.2 accuracy, CAREBENCH +3.1 action F1 and +6.9 object F1). The central comparison is against SFT on the same data, where RL wins even though SFT with structured thinking fails to help, which the authors attribute to teacher-forcing imitating format rather than learning the reasoning-to-caption link. If correct, this matters because it suggests reasoning-style RL can reduce the cost of improving captioning quality in video MLLMs.","feed_headline":"RL post-training lifts video caption accuracy with only 1.5k samples","feed_subtitle":"GRPO with think/caption rewards outperforms SFT on DREAM-1K, VDC, and CAREBENCH using just 1.5k samples.","key_machinery":"The load-bearing mechanism is a two-stage generation policy trained with GRPO under a composite reward: a format score, a think score, and a caption score. The think score (Tscore = 0.6×action F1 + 0.4×attribute F1) is obtained by solving bipartite matchings between predicted and annotated subjects, actions, and attributes, with similarity measured by the dot product of M3-Embedding vectors; the ground-truth annotation is itself generated by Qwen2.5-72B from source captions. The caption score is an event-coverage score: Qwen2.5-72B decides which ground-truth events are entailed by the generated caption, and the reward is 0, 0.5, or 1 depending on how much of the event list is covered. The composite reward makes the optimization signal decomposable: the model must learn to enumerate the right subjects and actions before it can collect the caption reward.","core_discovery":"VideoCap-R1 claims that the R1-style reinforcement-learning recipe transfers to open-ended video captioning when the task is decomposed into a structured thinking phase and a caption phase. The model first emits, inside a think tag, a subject-by-subject list of attributes and action sequences, then writes the final description in an answer tag. A think score, computed without an LLM by bipartite matching of action and attribute embeddings against structured annotations, rewards the reasoning content, while a caption score, computed by a Qwen2.5-72B judge through event-coverage entailment, rewards the final text. Training with GRPO on 1.5k curated videos improved event/action/object metrics on DREAM-1K, VDC, and CAREBENCH relative to both the base model and SFT-trained models; the authors further report that adding structured thinking to SFT degraded results, which they interpret as evidence that only RL establishes a genuine connection between reasoning and description. The caption-level gain on object F1 for CAREBENCH (34.3) surpasses GPT-4o-mini reported on the same benchmark.","pith_inferences":["If the reported gains hold under blind human evaluation, the same think/caption reward decomposition could transfer to other open-ended generation tasks, such as dense video grounding or embodied instruction following, where final outputs cannot be matched to a single ground truth.","The reward targets are generated by an LLM judge; our read is that the practical ceiling of VideoCap-R1 may be set by the alignment between Qwen2.5-72B's structured annotations and entailment judgments and human annotation, so the method should be re-run with human-written structured labels to test how much of the gain is judge-dependent.","Because the 1.5k training videos come from the Tarsier2-Recap-585K pipeline and DREAM-1K is built in the same lineage, a decontamination check is needed before concluding the DREAM-1K gain is purely capability rather than distribution overlap; this is our caution, not the paper's claim.","A cheap testable extension is to ablate the think prompt at inference: if the model is asked to final-answer only, the gains should shrink but not vanish if the RL-trained policy internalized the structured reasoning."],"forward_implications":["On the authors' evidence, GRPO with dual think/caption rewards is more data-efficient than SFT for improving video captioning: 1.5k samples beat SFT on the same budget across DREAM-1K, VDC, and CAREBENCH.","Adding structured thinking to SFT can hurt performance, so the benefit of reasoning-style data depends on the training algorithm, not just the data format.","A think reward that scores intermediate content is sufficient to shift the model's caption behavior, even when the final caption is scored by a separate signal.","The caption reward's event-coverage formulation reduces, in the paper's experiments, the reward hacking seen with direct LLM completeness/naturalness scoring.","Scaling this pipeline to larger training sets is the stated next step, implying the 1.5k-sample result is treated as a lower bound of feasible gains."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that the training pipeline is built on.","marker":"[28]"},{"why":"Establishes the R1 reinforcement-learning paradigm with rule-based rewards that the paper adapts to video captioning.","marker":"[11]"},{"why":"Provides the DREAM-1K benchmark and the event-coverage entailment prompt reused for the caption score.","marker":"[33]"},{"why":"Source of the Tarsier2-Recap-585K dataset from which the 1.5k training videos are sampled and curated.","marker":"[43]"},{"why":"The Qwen2.5-72B model that generates structured-thinking annotations and serves as the caption judge.","marker":"[38]"},{"why":"Supplies the M3-Embedding vectors used to compute similarity between predicted and ground-truth actions and attributes in the think score.","marker":"[4]"},{"why":"Defines the Qwen2-VL-7B baseline model that is fine-tuned and evaluated.","marker":"[34]"},{"why":"Provides the VDC benchmark and the VDCSCORE metric for evaluating detailed caption quality.","marker":"[3]"}],"fun_headline_variants":["Structured thinking plus RL boosts video captions with 1.5k samples","RL with think-then-answer beats SFT for video captioning","VideoCap-R1: GRPO improves video captioning via structured thought","1.5k samples enough for RL to improve video captioning","Structured thinking helps RL teach video MLLMs better actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward signals assume the Qwen2.5-72B-generated structured annotations and event-entailment judgments are faithful ground truth, so if those targets are noisy or biased, the policy could learn to please the judge rather than describe videos accurately.","fun_headline_variants_meta":{"raw":{"variants":["Structured thinking plus RL boosts video captions with 1.5k samples","RL with think-then-answer beats SFT for video captioning","VideoCap-R1: GRPO improves video captioning via structured thought","1.5k samples enough for RL to improve video captioning","Structured thinking helps RL teach video MLLMs better actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001431,"raw_usage":{"total_tokens":5816,"prompt_tokens":1032,"completion_tokens":4784,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":4689}},"tokens_in":648,"tokens_out":4784,"duration_ms":33156,"temperature":1.0,"reasoning_tokens":4689,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:35:05.581592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 100 held-out videos and have human annotators rate VideoCap-R1 captions against SFT and baseline captions for factual correctness; if human preference does not align with the reported F1/accuracy gains, the rewards are optimizing a proxy. A sharper version: retrain with human-written structured annotations instead of Qwen2.5-72B-generated ones and check whether the DREAM-1K event-F1 gain of +4.4 persists.","supporting_citations":[{"cited_title":"Auroracap: Efficient, performant video detailed captioning and a new benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the VDC benchmark and the VDCSCORE metric for evaluating detailed caption quality."}],"review_version":1}