{"id":"1d0d753d-4ab6-4ae7-98ee-6723c7eef741","arxiv_id":"2412.17637","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Introduces a sports commentary benchmark with 5,775 clips across six sports and a six-part GPT-based scoring metric, finding current video LLMs score poorly (best 5.44/10).","lead":"The paper builds a benchmark, SCBench, with 5,775 sports video clips and a six-part scorecard to test how well video AI models can generate commentary. It reports that current models score poorly, with InternVL-Chat-2 leading at 5.44 out of 10.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central model ranking rests on a GPT-4o-mini judge validated on only 15 samples with 60% agreement; that validation is too weak to rule out length/fluency bias, so the claim that all models fail at sports commentary is not yet supported.","rationale":"The reader's weakest assumption is exactly the validity of the GPT-4o-mini judge: only 15 human samples, 60% agreement, same model family as the label generator. My stress-test agrees and sharpens the concern with two specifics. First, the reported 15-sample agreement is not statistically established: 60% vs. 40% on 15 items is a difference of 3 samples, and no significance test or confidence interval is provided. Second, the paper's own observation that InternVL-Chat-2 produced notably longer outputs makes length bias a concrete alternative explanation for its 1.04-point lead; the 15-sample study cannot rule this out. If the judge is biased, then Tab. 3's rankings, the fine-tuning gains, and the central 'all models fail' conclusion all lose support. The dataset and task definition are still valuable, and the weakness is fixable by a larger human validation and by controlling for output length, so a conditional verdict is appropriate. I therefore recommend no change to the reader's CONDITIONAL verdict.","tokens_in":21482,"tokens_out":6410,"duration_ms":64808,"concrete_test":"Repeat the Sec. 5.3.1 human study on a stratified sample of at least 100 test clips with at least 3 independent annotators per clip, using majority vote as the human label. Compute SCORES-human agreement (Cohen's kappa) and compare it with vanilla-GPT and BLEU/CIDEr agreement using a paired bootstrap or McNemar test. If the 95% CI for the SCORES-minus-vanilla difference includes 0, or SCORES-human kappa is below 0.4, the current validation does not support the ranking; the verdict should remain conditional on such validation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's headline result—InternVL-Chat-2 at 5.44 vs. 4.40 for the runner-up, and the conclusion that current Video LLMs 'fail to perform successful sports commentary'—is produced entirely by a GPT-4o-mini judge (Sec. 3.3). The only human validation (Sec. 5.3.1, Fig. 7) uses 15 samples with 60% agreement between SCORES and human majority decisions, versus 40% for vanilla GPT. A 20-point gap on 15 items is not shown to be statistically significant; no confidence interval, kappa, or paired test is reported. The confound is concrete: Sec. 5.3.2 states that the winning model's outputs were 'notably longer,' so the judge may be rewarding verbosity or fluency rather than commentary accuracy. This risk is aggravated by the fact that GPT-4o-mini also generated the six-dimensional labels used both to prompt models during inference and to instruct the judge, so any systematic label or judge bias propagates into every score in Tab. 3. Without a larger, statistically powered human validation, the model ranking and the 'all models fail' claim are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SCBench, a benchmark for evaluating Video LLMs on sports commentary generation. It contributes CommentarySet, a dataset of 5,775 video clips across six sports with extracted broadcast commentary and GPT-4o-mini-generated six-dimensional labels, and SCORES, a six-dimensional evaluation rubric scored by a GPT-4o-mini judge. The authors evaluate eight open-source Video LLMs plus two fine-tuning strategies, report that InternVL-Chat-2 achieves the highest SCORES (5.44, 1.04 above the runner-up), and conclude that current Video LLMs are far from producing practical sports commentary. The benchmark addresses a genuinely underexplored task; however, the validity of the central ranking and the 'all models fail' conclusion depends on the adequacy of the GPT-based evaluation, which is not yet established.","tokens_in":21792,"tokens_out":2925,"duration_ms":29265,"significance":"If the evaluation is trustworthy, SCBench fills a real gap: it targets dense, temporally rich sports videos and open-ended commentary generation, departing from the QA-format benchmarks that dominate the field. The CommentarySet itself, with 5,775 clips across six sports and time-aligned broadcast commentary, is a potentially reusable resource, and the six-dimensional rubric is a reasonable attempt to capture aspects that BLEU/CIDEr miss. The authors should be credited for releasing the dataset plan and for explicitly attempting a human validation of the metric. However, the paper's headline claims all rest on a GPT-4o-mini judge whose agreement with human judgment is demonstrated on only 15 samples, and no statistical or robustness evidence is provided. The significance of the empirical conclusions is therefore conditional on substantial additional validation.","major_comments":[{"comment":"The human validation of SCORES uses only 15 samples and reports 60% agreement with human majority decisions versus 40% for vanilla GPT. No confidence interval, kappa, or significance test is given; a 9-of-15 versus 6-of-15 split is not statistically significant under a binomial or McNemar test. Since the entire model ranking in Tab. 3 is produced by this judge, the claim that SCORES is a meaningfully better metric than Vanilla GPT—and hence the validity of the benchmark—is unsupported. Please report the raw per-sample agreements, an appropriate test, and ideally a larger human study.","section":"§5.3.1, Fig. 7"},{"comment":"The same GPT-4o-mini model family generates the six-dimensional labels (Sec. 4.1, step 4) and serves as the judge (Sec. 3.3). Moreover, these labels are shown to models at inference time in the three-tier prompt and are also given to the judge to define the scoring criteria. Any systematic bias in GPT-4o-mini's label assignments therefore propagates into both model generation and evaluation, making the pipeline self-referential. The paper provides no human agreement measure on the label assignments themselves. Please provide a label-level validation (e.g., human agreement on a sample of labels) or an ablation that varies the label source (e.g., human labels on a subset) to show that the ranking is not an artifact of the label generator.","section":"§3.3, §4.1"},{"comment":"All SCORES results are reported as point estimates with no error bars, confidence intervals, or significance tests. Because the judge runs at temperature 0.1, repeated runs with different seeds would be straightforward to report. The claim that InternVL-Chat-2 'surpasses the second-best by 1.04' is a single realization of a stochastic judge; without variance information, the reader cannot assess whether a 1.04 gap is meaningful, particularly given that the judge may be sensitive to output length. Please report mean±std over multiple judge runs and, if possible, per-clip standard errors.","section":"Tab. 3, §5.2"},{"comment":"The paper itself observes that InternVL-2's outputs were 'notably longer' and interprets this as capturing more information. However, longer outputs are also exactly what a fluency- or verbosity-biased judge would favor. The 15-sample human validation is far too small to rule out this confound. A concrete test would be to compare human preference on a subset where outputs are matched for length, or to truncate InternVL-2's outputs to the length of the runner-up and re-score; without such an analysis, the top ranking and the broad 'all models fail' conclusion are not robust.","section":"§5.3.2"}],"minor_comments":[{"comment":"The abstract describes the dataset as 'professionally annotated sports video clips,' but the six-dimensional labels are generated by GPT-4o-mini (Sec. 4.1, step 4), not by professional annotators. Please align the wording with the actual annotation pipeline.","section":"Abstract and §4.1"},{"comment":"The Chat-UniVi-CoT row contains '3,00' instead of '3.00' in the soccer column; please fix the typo and check all numeric formatting.","section":"Tab. 3"},{"comment":"Several figures are numbered or captioned incorrectly (e.g., Fig. 10 appears twice, Fig. 14 and Fig. 15 both describe table tennis examples). Please renumber and deduplicate the supplementary figures.","section":"Supplementary, Figs. 10-16"},{"comment":"The evaluation section states that a temperature of 0.1 is used for GPT-4o-mini but does not specify how many judge calls are made per clip or whether the reported scores are averages over multiple calls; this should be stated explicitly.","section":"§5.1"},{"comment":"The paper claims that fine-tuning with ICL and CoT improves Video-LLaVA and Chat-UniVi by 8% and 6%, but these percentages are not clearly tied to the SCORES scale or to the numbers in Tab. 3; please clarify the computation (e.g., relative versus absolute improvement).","section":"§1 and §5.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark task and dataset are potentially valuable to the video-LLM community, but the empirical claims are currently resting on a validation that is too weak. I would urge the editor to require the additional statistical and robustness analyses outlined in the major comments before considering acceptance. The paper may also benefit from a more careful framing of what is 'professionally annotated' versus 'GPT-annotated' in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dataset and the six-dimensional rubric are genuinely useful, but I would not put weight on the model ranking or the 'all models fail' conclusion as they stand. The evaluation rests on a GPT-4o-mini judge whose only human validation is 15 samples, and the authors note that the top-scoring model produced notably longer outputs—exactly the kind of confound that breaks LLM-as-a-judge setups.\n\nWhat's actually new: CommentarySet aggregates 5,775 clips across six sports with professionally sourced commentary, and unlike prior sports caption datasets (TennisSet, FineGym, SoccerNet), it spans multiple sports and labels each clip along six dimensions (events, technique, background, tactics, match situation, emotion). That is a real gap in the benchmark landscape, and the label distribution analysis shows the dimensions do capture stylistic differences between sports. The data construction pipeline—audio extraction, textual and GPT-based merging, manual timestamp refinement—is described concretely enough to reproduce. The paper also goes beyond zero-shot evaluation by trying ICL and CoT fine-tuning, which is more than most benchmark papers bother with.\n\nSoft spots: the biggest one is the evaluation loop. GPT-4o-mini generates the six-dimensional labels, is used to prompt the models, and then serves as the judge. The same model family is scoring itself, effectively. The human study (Sec. 5.3.1, Fig. 7) is 15 samples with 60% overlap against human majority, versus 40% for vanilla GPT. That is suggestive, but with 15 samples the 20-point gap is not established; no confidence interval, kappa, or test is reported. And Sec. 5.3.2 says InternVL-2's outputs were 'notably longer,' which could easily push a fluency- or length-biased judge. Tab. 3 has no error bars, so the 1.04-point lead over the runner-up is not distinguishable from noise. The dataset is also not yet released, so the artifact can't be checked. These are fixable: run a larger human validation, ideally stratified by dimension, check judge score correlation with output length, and report variance across judge runs and model seeds. Minor: the cosine merging threshold (0.7) is given without sensitivity analysis.\n\nWho this is for: anyone building or evaluating Video LLMs on fine-grained temporal understanding. The dataset and rubric are worth using even if the current leaderboard is unreliable. I'd send it to reviewers, but with a clear request to address the judge validation before publication. If the authors fix that, this becomes a solid benchmark contribution.","headline":"The CommentarySet dataset is a real contribution, but the paper's leaderboard is not trustworthy yet—the judge is validated on only 15 samples and the winner's longer outputs raise a concrete bias concern.","tokens_in":22288,"tokens_out":2783,"would_cite":true,"duration_ms":25387,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper builds SCBench, a sports-commentary benchmark with 5,775 clips and a six-dimensional score, and reports that every tested video LLM fails at the task, with the best model (InternVL-Chat-2) reaching only 5.44/10.","keywords":["sports video commentary","video large language models","benchmark","six-dimensional metric","GPT-based evaluation","CommentarySet","SCORES","temporal video understanding"],"falsifier":"Score the same model outputs with a different judge model or with outputs normalized by length; if InternVL-Chat-2 no longer leads, or if a larger human study drops agreement with SCORES well below 60 percent, the ranking and the 'all models fail' conclusion become an artifact of the judge.","tokens_in":21327,"feed_emoji":"🎙️","tokens_out":8153,"duration_ms":63338,"temperature":0.7,"pith_summary":"This paper claims that sports commentary is a hard, realistic test for video language models, harder than captioning or QA benchmarks, and that current open-source models all fail it. To make the test, the authors assemble CommentarySet, 5,775 professionally commented clips across six sports, and SCORES, a six-dimensional rubric (key events, technical detail, background, tactics, match situation, emotion) scored by a GPT judge against ground truth. Their central result is that InternVL-Chat-2 performs best at 5.44 out of 10, beating the next model by 1.04, and that fine-tuning with chain-of-thought improves two small models by 8% and 6% but still leaves them far from human-level commentary. The paper's significance, if true, is that fluent video understanding is not enough: models need fine-grained temporal and contextual understanding plus professional narration, and current benchmarks do not measure this.","feed_headline":"All video AI models fail at sports commentary, benchmark finds","feed_subtitle":"InternVL-Chat-2 tops the field at 5.44 of 10, 1.04 ahead of the runner-up.","key_machinery":"The engine of the benchmark is CommentarySet, a dataset of 5,775 short video clips (about 14.5 seconds each) with real broadcast commentary as ground truth, plus GPT-generated labels that partition each commentary into one or more of six dimensions: key events, technical detail, background information, tactical analysis, match situation, and emotional expression. The evaluation mechanism is SCORES, a prompt-based protocol in which GPT-4o-mini is first given the task definition, then the six-dimensional metric definitions, and then asked to grade model commentary against the ground-truth commentary on a 0-10 scale. The three-tier prompt structure (task definition, metric definition, task requirement) is what turns a generic LLM judge into a domain-specific one; the same dimension definitions are also fed to video models during inference to channel generation.","core_discovery":"The central claim, stated on the paper's own terms, is that sports commentary generation is a valid and demanding benchmark for video LLMs, and that every open-source model tested fails at it. The paper constructs CommentarySet, 5,775 professionally commented video clips across athletics, basketball, soccer, gymnastics, table tennis, and tennis, with ground-truth commentary and six-dimensional labels. It then introduces SCORES, a metric that scores generated commentary on key events, technical detail, background information, tactical analysis, match situation, and emotional expression, administered through a GPT judge that compares model output with ground truth. On this benchmark, InternVL-Chat-2 scores 5.44 out of 10, 1.04 points above the runner-up Mini-InternVL-Chat-v1.5, and the authors interpret the uniformly mediocre scores as evidence that current models cannot perform practical sports commentary. They also report that fine-tuning Video-LLaVA and Chat-UniVi with chain-of-thought improves their scores by 8% and 6%, respectively, but leaves them below several zero-shot models, indicating that stylistic alignment alone is insufficient.","pith_inferences":["A testable extension not in the paper: re-score the 867 test clips with a different judge model or a length-controlled version of GPT-4o-mini; if the ranking changes, the headline result is partly an artifact of judge preference for longer outputs, which the paper itself notices InternVL-2 tends to produce.","The six-dimensional label distribution per sport is effectively a commentary-style profile; one could build a style-conditioned generator that takes a desired dimension mix as input, which the paper mentions only as a possibility.","Because the ground-truth labels and the judge are generated by the same model family, the benchmark may partially measure self-consistency; a cross-family judge study would clarify how much of the score reflects genuine commentary quality.","The dataset could be re-purposed for image LLMs by sampling frames, as the authors note, which would let the same benchmark compare video-native and frame-based models."],"forward_implications":["If the benchmark is accepted, future video LLM comparisons will need a generation task with temporal and emotional dimensions, not just QA, to expose real-world capability gaps.","The 8% and 6% gains from chain-of-thought fine-tuning quantify how much of the deficit is style and prompt alignment, leaving the rest attributable to visual understanding and knowledge.","The 1.04-point gap between InternVL-Chat-2 and the runner-up becomes a concrete target for other developers.","Current automatic captioning metrics (BLEU, CIDEr) are shown to be near-useless for this task, pushing evaluation toward LLM-based judges with structured rubrics.","Because the dataset includes per-dimension labels, future work can train or prompt models to emphasize specific commentary dimensions rather than all at once."],"supporting_citations":[{"why":"FineGym supplies part of the gymnastics clips in CommentarySet.","marker":"Shao et al. (2020)"},{"why":"TenniSet supplies tennis clips and is the prior single-sport dataset the benchmark extends.","marker":"Faulkner and Dick (2017)"},{"why":"Video-LLaVA is one of the two base models fine-tuned and a zero-shot baseline.","marker":"Lin et al. (2023)"},{"why":"Chat-UniVi-1.5 is the other base model fine-tuned with ICL and CoT.","marker":"Jin et al. (2024)"},{"why":"VILA serves as a zero-shot baseline at 34B scale in the evaluation.","marker":"Lin et al. (2024)"},{"why":"InternVL-Chat-2 is the paper's best-performing model, scoring 5.44.","marker":"InternVL-2 Team"},{"why":"Mini-InternVL-Chat-v1.5 is the runner-up at 4.40 and a zero-shot baseline.","marker":"Gao et al. (2024)"},{"why":"ICL is the in-context-learning fine-tuning baseline.","marker":"Brown et al. (2020)"},{"why":"Chain-of-thought is the prompting/fine-tuning method compared against ICL.","marker":"Wei et al. (2023)"},{"why":"BLEU is one of the traditional captioning metrics that SCORES is shown to outperform.","marker":"Papineni et al. (2002)"}],"fun_headline_variants":["Sports commentary benchmark stumps video AI","Video AI fails sports commentary test, best score 5.44","New benchmark: No video LLM passes sports commentary","Top video model scores 5.44/10 on sports commentary benchmark","Sports commentary exposes video AI blind spots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4o-mini's six-dimensional scores agree with human judgment well enough to rank models, a check done on only 15 samples with 60 percent agreement.","fun_headline_variants_meta":{"raw":{"variants":["Sports commentary benchmark stumps video AI","Video AI fails sports commentary test, best score 5.44","New benchmark: No video LLM passes sports commentary","Top video model scores 5.44/10 on sports commentary benchmark","Sports commentary exposes video AI blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2123,"prompt_tokens":1097,"completion_tokens":1026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":949}},"tokens_in":713,"tokens_out":1026,"duration_ms":9665,"temperature":1.0,"reasoning_tokens":949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:19:56.690368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same model outputs with a different judge model or with outputs normalized by length; if InternVL-Chat-2 no longer leads, or if a larger human study drops agreement with SCORES well below 60 percent, the ranking and the 'all models fail' conclusion become an artifact of the judge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TenniSet supplies tennis clips and is the prior single-sport dataset the benchmark extends."},{"cited_title":"B leu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"BLEU is one of the traditional captioning metrics that SCORES is shown to outperform."}],"review_version":1}