{"id":"a6da1122-307a-4342-83f3-c642f8faed96","arxiv_id":"2608.06930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An audio-video captioning model trained on a new 100K dataset with a detail-aware GRPO reward, plus a matching atomic QA benchmark, reports gains over open-source baselines and competitive results with Gemini-2.5-Pro on several evaluations.","lead":"This paper introduces AVCap-100K, a 100K-video dataset of fine-grained audio-visual captions, and trains AVCap, a model that uses a new dense reward (Da-GRPO) to improve caption detail and reduce hallucination. It also proposes a benchmark and metric, AVCap-Bench and AVCap-Score, for evaluating atomic visual, audio, and joint details, reporting gains over open-source baselines and competitive results with Gemini-2.5-Pro on several tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AVCap-Score may reward the judge's priors rather than caption content; no negative-control test shows answer extraction is actually grounded in the caption.","rationale":"The reader's weakest assumption was AVCap-Score validity, centered on style bias from the same model family. The concern raised here is a sharper, more directly load-bearing version: the metric's three-stage pipeline (Eqs. 1-3 and Section 3.3) assumes the judge answers only from the provided caption, and no test in the paper checks this by removing or contradicting facts. This matters because Da-GRPO optimizes this exact score; if the judge can answer from priors, the reward is not dense or detail-aware, and AVCap-Bench results are inflated. I do not call for rejection: Video-SALMONN-2, UGC-VideoCap, and DailyOmni/WorldSense use independent evaluation protocols and consistently favor AVCap, and the B.5 human evaluation supports the model-level ranking. Those results keep the paper's practical contribution viable, but they do not validate the 'atomic-level, strict' metric or the reward's grounding, so the conditional status remains. The proposed negative-control test would settle the concern directly and should be a release requirement.","tokens_in":27202,"tokens_out":5428,"duration_ms":63493,"concrete_test":"Run a negative-control experiment on 100 AVCap-Bench samples using the exact pipeline of Appendix C.3. For each sample, build four caption variants: (a) original, (b) one atomic fact deleted, (c) all probed facts replaced with plausible incorrect facts, (d) an unrelated caption containing none of the facts. Compute AVCap-Score for all variants with Qwen3-30B-A3B at temperature 0, and have three independent human annotators rate the variants by factual detail. If scores for (b), (c), or (d) do not drop sharply (e.g., by more than 80% of the original score) or if extracted answers still match gold answers, the judge is answering from priors rather than the caption, invalidating both the reward and the metric; report per-sample Spearman correlation with the human ratings as the validity evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two coupled parts: Da-GRPO's reward 'strictly measures whether the predicted caption accurately preserves fine-grained details' (Section 3.2), and AVCap-Score quantifies 'informational equivalence' (Section 3.3). Both rely on Eq. (1): the judge model Mjudge answers probe q_k with the candidate caption as 'Context'. If Mjudge can produce a correct answer from its own video/commonsense priors even when the caption omits or contradicts the fact, then the score does not measure recoverability from the caption, and the reward can be optimized by writing 'Qwen-like' captions rather than more accurate ones. The paper never runs this negative control: Table A6 audits only 100 judge answers where the fact is present (error rate 1%), and Appendix C.2 swaps only the scoring model, not question generation or answer extraction, so the same-family bias remains. Because AVCap-Score is also the optimization target, a positive result on AVCap-Bench is partly circular unless the metric is validated against human judgments of factual detail. External benchmarks and the B.5 human evaluation provide independent support for the model, but they do not validate the 'atomic-level, strict' metric that the paper's headline contribution rests on.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AVCap, a framework for detailed audio-video joint captioning, comprising three contributions: (1) AVCap-100K, a 100K-video dataset of temporally aligned, detail-rich captions generated via a multi-stage pipeline with explicit visual/audio disentanglement and rating-based filtering; (2) Da-GRPO, a GRPO variant whose reward is built from atomic QA probes derived from ground-truth captions, with the judge model answering each probe from the candidate caption and scoring semantic similarity to the gold answer; and (3) AVCap-Bench and AVCap-Score, a benchmark and metric that evaluate atomic-level factual preservation using the same probe-and-verify paradigm. The authors claim that AVCap-7B-SFT outperforms open-source baselines of similar scale and that AVCap-30B reaches 56.94 on AVCap-Bench, 85.1 on UGC-VideoCap, and 32.7 on Video-SALMONN-2, remaining competitive with Gemini-2.5-Pro on several evaluations.","tokens_in":27395,"tokens_out":6018,"duration_ms":58126,"significance":"If the core claims hold, the framework is valuable to the community: it releases a large-scale fine-grained audio-visual caption dataset, a dense reward design that appears to improve factual detail over coarse rewards, and an atomic-level evaluation metric. The paper is commendably reproducible, with code, models, and datasets released, and it provides controlled ablations (Tables 3 and 4) showing consistent gains on both a 7B dense model and a 30B-A3B MoE model. The external benchmarks (Video-SALMONN-2, UGC-VideoCap, DailyOmni, WorldSense) and the human evaluation in Appendix B.5 offer independent support for the model's overall quality. The main unresolved issue is the validity of AVCap-Score as an independent measure of fine-grained factual fidelity, which is load-bearing for the paper's central claim that Da-GRPO 'strictly measures' atomic-level detail preservation.","major_comments":[{"comment":"AVCap-Score is both the training reward (Eq. 3) and the headline evaluation metric (Section 3.3), and both use judge models from the Qwen3 family that also generated the ground-truth captions in AVCap-100K. The paper does not provide a negative control showing that the judge's answer to a probe is actually grounded in the provided caption. The audit in Table A6 checks 100 judge answers where the fact is present (1% error rate), but it does not test the case where the caption omits or contradicts the fact, which is exactly the case where a prior-driven judge would still produce a high score. The claim in Section 3.2 that the reward 'strictly measures whether the predicted caption accurately preserves the fine-grained visual and auditory details' is therefore not yet established; a style-matching or judge-prior explanation remains plausible. I recommend adding a controlled perturbation study (e.g., removing or altering the fact-bearing sentence in the caption and measuring whether the probe score drops) and, if possible, correlating AVCap-Score with human judgments of atomic fact preservation on a held-out subset.","section":"Section 3.3 and Eq. (3)"},{"comment":"The alternative-judge experiment swaps only the scoring model (Gemini-3.1-Pro and GPT-5.4), but question generation and answer extraction still use Qwen3-30B-A3B. The probe questions are therefore generated by a model from the same family as the trained model and as the data-generation model, and the ranking of Qwen-family models may still be inflated by shared stylistic priors. While the relative ranking of AVCap-30B over baselines is preserved under the alternative judges, this experiment does not address the core circularity concern because the full probe-generation pipeline is unchanged. To properly test for judge-family bias, the question generation and answer-extraction steps should also be swapped, or the metric should be validated against human-annotated atomic facts on the same 1,000-video benchmark.","section":"Appendix C.2, Table A9"},{"comment":"There is an inconsistency between the description of the reward model and the evaluation judge. Section 3.2 refers generically to the 'Judge Model Mjudge' for reward computation, while Appendix A.4, Table A4 specifies that the reward node in Da-GRPO uses Qwen3-14B with thinking disabled. Evaluation in Section 3.3, in contrast, uses Qwen3-30B-A3B-Instruct as the judge. The paper does not discuss whether this reward/evaluation judge mismatch affects the alignment between the training target and the evaluation metric. This is a concrete reproducibility and interpretation issue, and the authors should either use the same judge for both or report ablations showing that the choice of reward judge does not materially change the results.","section":"Section 3.2, Eq. (1)-(3) and Appendix A.4, Table A4"},{"comment":"The paper states that AVCap-Score 'directly measures whether the atomic fact queried by qk is recoverable from the generated caption,' but no experiment directly tests recoverability. The current evidence consists of a manual audit of 100 generated probe-answer pairs (Table A6) and the human evaluation (Appendix B.5), but the latter scores agreement with the ground-truth caption, not the metric's sensitivity to fact omission. I recommend adding a sensitivity analysis where candidate captions are systematically perturbed (e.g., deleting audio facts, visual facts, or joint claims) and showing that AVCap-Score drops accordingly. This would substantiate the 'strict' claim and would also make the metric more useful to the community.","section":"Section 3.3, Question Generation Principles and Score Calculation"}],"minor_comments":[{"comment":"Typo: 'exsisting' should be 'existing' in the third paragraph of the Introduction.","section":"Section 1"},{"comment":"Typo: 'insted' should be 'instead' in the Visual Filtering User Prompt.","section":"Appendix B.6.1"},{"comment":"The caption says 'accuracy of atomic fact verification,' but the scores are semantic-similarity scores, not binary accuracy. Please rephrase to avoid confusion.","section":"Table 2 caption"},{"comment":"The notation 'Qv', 'Qa', 'Qav' is used for question subsets, but later 'Q' is reused for the full probe set. Consider using distinct symbols (e.g., 'Q_v', 'Q_a', 'Q_av', and 'Q_total') to reduce ambiguity.","section":"Section 3.3"},{"comment":"The equation writes 'AVCap-Score = Σ Score_i = 87.0', but the example in Table A10 sums the 20 per-question scores. The relationship between these per-question scores and the modality-normalized sub-scores defined in Section 3.3 is not explicitly shown; adding a one-line derivation would improve reproducibility.","section":"Appendix C.3.5, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper is generally well-executed and the ablations are reassuring, but the central metric requires validation against an independent ground truth before the 'atomic-level, strict' claim can be taken at face value. The authors would benefit from a negative-control experiment showing that AVCap-Score is sensitive to fact removal, and from a human-validated subset of the benchmark. If these additions are made, the contribution is likely to be a solid and citable resource for the audio-video captioning community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a genuine engineering contribution with a large new dataset, a sensible dense-reward RL recipe, and a reusable QA-style evaluation setup. The strongest claims about AVCap-Score being an atomic-level, strict measure of factual preservation are not yet backed by evidence, but the underlying model improvements look real.\n\nWhat is actually new: AVCap-100K, a 100K video-caption dataset with a multi-stage annotation pipeline; Da-GRPO, a GRPO variant that uses judge-generated atomic probes as a dense reward; and AVCap-Bench/Score, a QA-based metric for fine-grained audiovisual caption evaluation. The ablations in Tables 3 and 4 are the most convincing part: Da-GRPO consistently beats SFT, holistic reward, and n-gram reward at both 7B and 30B scale, and the dataset alone gives solid gains. External benchmarks (UGC-VideoCap, Video-SALMONN-2, DailyOmni, WorldSense) provide some independent evidence, and the small human evaluation points the same direction. So there is a real contribution here.\n\nThe soft spots are concentrated in the evaluation loop. The same Qwen3 family generates the training captions, writes the probe questions, answers them from candidate captions, and grades the answers. That means AVCap-Score could reward captions that look like Qwen-style writing rather than captions that actually preserve video facts. The paper's audit checks only 100 judge answers where the fact is present, so it cannot catch the judge silently filling in missing facts from priors. There is no negative-control test. Appendix C.2 swaps only the scoring model, not question generation or answer extraction, so the same-family bias remains. Also N=20 and the 5/5/10 question mix are tuned on the authors' own metric; filtering thresholds are undocumented; and there are no error bars or significance tests. These issues do not sink the model-level claims, because the external benchmarks and human eval are independent, but they do mean the headline AVCap-Score numbers should not be read as a validated measure of factual detail.\n\nThe citation pattern is fine. The paper builds on GRPO and LLM-judge QA evaluation, cites the relevant precedents, and the self-citations are mostly to prior work from the same group that is directly on topic.\n\nWho this is for: researchers working on detailed video captioning, RL post-training for multimodal models, and QA-based automatic metrics. It deserves a serious referee. A good review would push for cross-family or human validation of AVCap-Score, a negative-control experiment to test whether answers are actually grounded in the caption, multi-run statistics, and full release of data, code, and filtering thresholds. If those land, this becomes a useful subfield-level advance.","headline":"A solid dataset-and-RL package with real gains on external benchmarks, but the headline AVCap-Score metric is not yet validated independently, so the 'strict atomic-level' claims should be taken as provisional.","tokens_in":28067,"tokens_out":2095,"would_cite":true,"duration_ms":23578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dense question-answer rewards sharpen audio-video captions to match proprietary models.","keywords":["audio-video captioning","detail-aware reward","GRPO","atomic-level evaluation","video captioning dataset","multimodal large language models","audiovisual alignment","caption quality metric"],"falsifier":"Inject controlled factual errors into captions, such as changing an instrument, swapping a color, or shifting an audio-visual timing, and compare AVCap-Score deltas with human error counts; if score changes do not monotonically track the number and severity of injected errors, the atomic-factuality claim of the metric fails.","tokens_in":26930,"feed_emoji":"🎬","tokens_out":5725,"duration_ms":56529,"temperature":0.7,"pith_summary":"AVCap argues that fine-grained audio-video captioning is limited by reward and evaluation granularity as much as by data scarcity. To break that limit, it builds AVCap-100K, a 100K-video dataset of temporally aligned, detail-rich captions, and trains captioners with a new reward, Da-GRPO, that grades captions on atomic visual, audio, and joint facts instead of a single holistic score. It also proposes AVCap-Bench and AVCap-Score, a 1,000-video benchmark and a probe-based metric that measures whether atomic facts survive in a generated caption. The paper reports that the trained AVCap-30B reaches 56.94 on AVCap-Bench, 85.1 on UGC-VideoCap, and 32.7 on Video-SALMONN-2, matching or surpassing Gemini-2.5-Pro on several evaluations. If right, it gives the community a dataset, reward design, and evaluation standard for detailed captioning.","feed_headline":"Dense rewards push open caption models past Gemini on details","feed_subtitle":"A new 100K dataset and atomic probe-based metric let a 30B model match or beat proprietary rivals on fine-grained captions.","key_machinery":"The load-bearing mechanism is the Da-GRPO reward: a judge model raises N atomic probes from the ground-truth caption, answers each probe from the candidate caption alone, and scores semantic similarity against ground-truth answers; the normalized average becomes the reward. This turns caption quality into a dense verification problem rather than a single holistic judgment. The same probe-and-score protocol is reused as AVCap-Score, making the training reward and the evaluation metric share the same atomic fact representation.","core_discovery":"The central claim is that replacing coarse, event-level reward signals with a dense, atomic-level verification signal materially improves factual detail in audio-video captioning. Da-GRPO works by having a judge model raise a set of granular questions about the ground-truth caption, split into visual, audio, and audio-visual joint facts, answer those questions using only the generated caption as context, and then grade the semantic similarity of the answers. The averaged similarity across the probe set is the reward fed into group-relative policy optimization. The paper contends that this Raise-Answer-Check reward is what lets a 7B model surpass existing open-source baselines and a 30B model match or beat Gemini-2.5-Pro on several evaluations, and that the same probe protocol, applied as AVCap-Score, gives the first atomic-level evaluation standard for audiovisual captions.","pith_inferences":["Since the judge model both produced the training captions and grades the benchmark answers, part of the measured gain may reflect style matching to the judge's preferred caption form; a human study comparing AVCap-Score against human factual-error counts would tell how much of the gain is truly factual.","The atomic QA reward is a general recipe: any generator whose output can be probed by question-answer fact checks could be trained with the same Raise-Answer-Check loop, including image captioning, long-form video grounding, or fact-checking pipelines.","The 100K captions, with mean length around 416 words and explicit audio-visual alignment, could double as conditioning data for text-to-video generation, where dense joint descriptions are known to help controllability."],"forward_implications":["A 7B audio-video model trained on AVCap-100K with SFT alone surpasses prior open-source baselines, separating the dataset's contribution from the RL stage's.","Adding Da-GRPO to the SFT checkpoint improves all reported benchmarks at both 7B and 30B scales, showing that reward granularity, not just data volume, drives fine-grained caption quality.","Because AVCap-Score grades recoverable atomic facts, captioners trained with it should show fewer hallucinations and omissions on details such as colors, instrument names, accents, and audio-visual synchronization.","The released dataset and metric give future work a shared target: a 100K dense-caption resource and a reproducible judge-based evaluation protocol."],"supporting_citations":[{"why":"Base RL algorithm (GRPO) that Da-GRPO extends with the dense atomic reward.","marker":"[16]"},{"why":"Prior state-of-the-art audio-video captioner with temporal orchestration; supplies the main baseline and evaluation protocol on Video-SALMONN-2.","marker":"[1]"},{"why":"UGC-VideoCaptioner introduces the UGC-VideoCap benchmark and coarse event-level RL reward that Da-GRPO is compared against.","marker":"[2]"},{"why":"Video-SALMONN-2 test set used for zero-shot evaluation of general audio-visual understanding.","marker":"[12]"},{"why":"Demucs source separation splits vocal and background tracks, enabling the hierarchical audio annotation branch.","marker":"[37]"},{"why":"Provides the fact-checking and question-probe idea that Da-GRPO's reward and AVCap-Score inherit.","marker":"[44]"},{"why":"Gemini-2.5-Pro is the proprietary model whose scores AVCap claims to match or surpass on several evaluations.","marker":"[17]"},{"why":"Qwen2.5-Omni backbone used for the 7B model and a baseline in comparisons.","marker":"[22]"}],"fun_headline_variants":["Atomic rewards push open audio-video captions past Gemini","Dense probe rewards let 30B open model surpass Gemini on AV captions","Raise-Answer-Check reward lifts AV captions to new heights","Detail-aware rewards make open models match Gemini on AV captions","Atomic probe reward drives AV captioning to SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's validity rests on the judge model scoring semantic similarity in a way that tracks factual alignment rather than stylistic similarity to the judge's own preferred captions, and the paper does not validate this assumption against human judgments of the metric itself.","fun_headline_variants_meta":{"raw":{"variants":["Atomic rewards push open audio-video captions past Gemini","Dense probe rewards let 30B open model surpass Gemini on AV captions","Raise-Answer-Check reward lifts AV captions to new heights","Detail-aware rewards make open models match Gemini on AV captions","Atomic probe reward drives AV captioning to SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001007,"raw_usage":{"total_tokens":4244,"prompt_tokens":922,"completion_tokens":3322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":3233}},"tokens_in":538,"tokens_out":3322,"duration_ms":25636,"temperature":1.0,"reasoning_tokens":3233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:08:50.453077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inject controlled factual errors into captions, such as changing an instrument, swapping a color, or shifting an audio-visual timing, and compare AVCap-Score deltas with human error counts; if score changes do not monotonically track the number and severity of injected errors, the atomic-factuality claim of the metric fails.","supporting_citations":[{"cited_title":"Hybrid transformers for music source separation","cited_arxiv_id":null,"evidence_quote":"Demucs source separation splits vocal and background tracks, enabling the hierarchical audio annotation branch."},{"cited_title":"Auroracap: Efficient, performant video detailed captioning and a new benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the fact-checking and question-probe idea that Da-GRPO's reward and AVCap-Score inherit."}],"review_version":1}