{"id":"3251c9d9-c80f-40f3-9c9d-122dd0695bff","arxiv_id":"2506.11155","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoCaption uses MCTS to generate fine-grained video key points, forming the MCTS-VCB benchmark that ranks MLLMs and yields training data improving a fine-tuned model's captioning.","lead":"This paper presents AutoCaption, a Monte Carlo Tree Search based system that automatically generates thousands of detailed descriptive key points per video, and uses them to build a new video captioning benchmark called MCTS-VCB. The benchmark is used to evaluate over 20 multimodal LLMs, and the generated data also improves a fine-tuned model's captioning performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-check pass rate (94.7%) is reported for the 0.9-threshold key point set, but all experiments use the 0.8-threshold set; if the 0.8 set was not human-verified, ground-truth errors could distort every ranking and fine-tuning gain.","rationale":"The reader's weakest assumption is the most load-bearing: the automatic verification pipeline and ambiguous human-check coverage. My independent reading finds the same. The paper has genuine strengths: the human-consistency correlations in Table 2 (up to 88.5 Pearson) support the evaluation metric's ranking validity; the out-of-domain DREAM-1K fine-tuning gain (16.3%) is nontrivial even if the in-domain benchmark were flawed; the code/data release enables audits; and the MCTS-vs-beam-search ablation (91.62% vs 83.43% mutual inclusion) provides some evidence for the MCTS contribution. However, the correctness of the benchmark ground truth is the foundation on which the ranking claim, the fine-tuning claim, and the MCTS-vs-beam-search comparison all rely. If the final 0.8-threshold key point set was not human-verified, the 94.7% pass rate does not validate it, and the automated verifiers' potential self-bias (GPT-4o as verifier and evaluated model) is unmitigated. A targeted release and audit of human labels would settle this; until then a conditional verdict is appropriate.","tokens_in":31010,"tokens_out":12067,"duration_ms":124410,"concrete_test":"Ask the authors to (1) specify whether the 0.8-threshold set is a subset of the human-checked 0.9 set and (2) release per-key-point human labels from the check. Then recompute Table 1 overall F1 rankings and Table 4 fine-tuning gains using only key points that are known to have passed human verification, or, if no such labels exist, run a fresh human audit on a random sample of 200 videos from the released 0.8-threshold key points. If the model rankings and the +25.0% (MCTS-VCB) and +16.3% (DREAM-1K) gains are unchanged on the human-verified subset, the concern is resolved; if they shift materially, the automated verification, not human-checked ground truth, is driving the central claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.4 states that annotators checked key points 'generated under a similarity threshold of 0.9' and that the human verification pass rate was 94.7%, while Section B.3 states that all experimental results are based on a similarity threshold of 0.8. The paper never shows that the 0.8-threshold key points used in Tables 1–4 and Figure 4 are a subset of, or otherwise covered by, the human-checked 0.9 set. If the final set was not human-verified, the only quality filter is the automated two-MLLM yes/no verification (GPT-4o and Qwen2-VL-72B, with GPT-4-generated questions). That automated filter is exactly where systematic bias can enter: GPT-4o both verifies key points and is one of the evaluated models, so false key points matching GPT-4o's own captioning tendencies could be preferentially retained and then scored as correct for GPT-4o. The same applies to Gemini-1.5-Pro, which contributes the root 'Overall Description' and is then evaluated. Because the benchmark's central contribution is to rank MLLMs and to demonstrate fine-tuning gains, an unverified or biased ground-truth set would invalidate those claims. The ambiguity is compounded by inconsistent pipeline descriptions: Section 4.1 says Qwen2-VL-72B extracts key points during verification, while Appendix A.3 says GPT-4 extracts them. This makes it impossible to fully audit the ground-truth construction from the paper alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AutoCaption, an automated framework that uses Monte Carlo Tree Search (MCTS) to iteratively generate fine-grained descriptive key points from videos, and applies it to build MCTS-VCB, a video captioning benchmark containing 1,765 videos with on average 122.3 key points per video. The authors evaluate 21 open- and closed-source MLLMs on MCTS-VCB, report human-consistency correlations for a subset of models and videos, and show that fine-tuning InternVL2.5-8B on AutoCaption-generated data improves its F1 by 25.0% on MCTS-VCB and 16.3% on DREAM-1K, surpassing fine-tuning directly on GPT-4o-generated captions.","tokens_in":31425,"tokens_out":5300,"duration_ms":60322,"significance":"If the claimed results hold, MCTS-VCB would provide a much denser and more diverse key-point resource than prior video captioning benchmarks such as DREAM-1K, and AutoCaption would offer a scalable and relatively low-cost way to build both evaluation benchmarks and training data. The paper is also valuable for releasing code and data, and for reporting human-consistency correlations that go beyond simple n-gram metrics. However, the benchmark's validity depends critically on the reliability of the automated key-point verification pipeline and on the consistency between the human-checked data and the data actually used in the experiments; these points are currently not established in the manuscript.","major_comments":[{"comment":"The human-check pass rate of 94.7% is reported for key points generated under a similarity threshold of 0.9, yet all experimental results, including Tables 1, 2, 4 and Figure 4, are stated in Appendix B.3 to be based on a similarity threshold of 0.8. The paper never demonstrates that the 0.8-threshold key points used in the evaluation are identical to, or directly derived from, the 0.9-threshold set that annotators checked. If the 0.8 set was not human-verified, its only quality filter is the automated two-MLLM yes/no verification, and ground-truth errors could distort every model ranking and the fine-tuning gains reported in Section 6.2. The authors should either human-check a random sample of the final 0.8-threshold key points, or explicitly prove that the human-checked 0.9 set covers the 0.8 set, and report the verification results for the actual evaluation set.","section":"Appendix A.4 and Appendix B.3"},{"comment":"The verification pipeline is potentially circular: GPT-4o and Qwen2-VL-72B are used as the two verifiers that admit key points into MCTS-VCB, and both models are themselves evaluated on MCTS-VCB (Table 1). In addition, Gemini-1.5-Pro generates the initial Overall Description that seeds the search tree, and it is also evaluated and ranks first overall. This creates a concrete correctness risk: if a verifier systematically over-accepts statements that match its own captioning tendencies, the benchmark ground truth becomes biased toward that verifier's style, inflating its score and potentially affecting rankings of the other models. I am not claiming deliberate bias, but the paper should address this by reporting per-verifier pass rates and disagreement statistics, and by showing that rankings remain stable after removing key points that are admitted by only one of the two verifiers or when using a third, unevaluated verifier.","section":"Section 4.1 and Section 6.1 / Table 1"},{"comment":"There is a direct inconsistency in the description of the key-point extraction step. Section 4.1 states that key points are extracted from a node by Qwen2-VL-72B under the prompt in Figure 13, while Appendix A.3 states that GPT-4 is used to extract key points from the node's action results with the same Figure 13 prompt. This matters because the same extraction prompt is later used in Section 6.1 to obtain KP_model from model-generated captions. Without knowing which model actually extracted key points during benchmark construction, the pipeline cannot be fully audited, and the potential for Qwen-family bias in the benchmark cannot be assessed. The authors should unify the description and clearly specify which model was used at each stage.","section":"Section 4.1 vs. Appendix A.3"}],"minor_comments":[{"comment":"In the paragraph 'Performance on different frame numbers', the text refers to 'Figure 3' when reporting the frame-number results; the relevant results appear in Table 3, not in Figure 3 (which illustrates key-point verification). Please correct the cross-reference.","section":"Section 6.1"},{"comment":"There are numerous typographical and grammatical errors, including 'tranditional', 'excuating', 'MMLM', 'cantains', and 'defination'. A thorough language edit would improve readability.","section":"Throughout"},{"comment":"The values of alpha and beta in Q(s,a) are set to 0.5 based on the claim that MC(s) and SM(s) are 'same important', but no sensitivity analysis is provided for these weighting parameters or for the PUCT constant c. Since these hyperparameters affect which nodes are expanded, a small robustness study would strengthen the claim that the benchmark is stable to design choices.","section":"Section 4.1, Eq. (2)"},{"comment":"The appendix section heading is 'A.3 MCTS Evaluation Phase Details', but the first sentence of the final paragraph says 'Ultimately, our method improves upon Method 2...' without numbering the proposed method as Method 3. Minor renumbering would make the comparison clearer.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The central idea is interesting and the released resource could be useful, but the manuscript currently has a load-bearing transparency gap: the human-checked data threshold differs from the threshold used in all experiments, the verification pipeline involves models that are themselves evaluated, and two sections contradict each other about which model extracts key points. These are fixable without changing the paper's scope, but they must be addressed before the benchmark can be trusted as a ranking instrument. I saw no evidence of intentional misconduct; the issues appear to be omissions in reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine new contribution—MCTS as a key point generator for video captioning benchmarks, a new 1,765-video benchmark with 122 key points per video, and a fine-tuning recipe that lifts InternVL2.5-8B by double digits on both in- and out-of-domain sets. If the ground truth holds up, it is a useful evaluation resource. But the ground-truth audit has a hole the authors need to close before I'd trust the rankings.\n\nWhat's good: The MCTS iteration idea is sensible, the six action types cover the right dimensions, and the ablations (MCTS vs beam search, similarity threshold, frame counts, human consistency) are more thorough than most benchmark papers. They ship code and data. The human consistency study on 100 videos gives reasonable evidence the metric tracks human judgment. The fine-tuning comparison against GPT-4o captions is a fair baseline and the AutoCaption data wins.\n\nWhere it goes soft: The verification pipeline is the load-bearing wall. Key points are verified by two MLLMs (GPT-4o and Qwen2-VL-72B) answering yes/no questions generated by GPT-4. Both GPT-4o and Gemini-1.5-Pro are used in construction (Gemini co-writes the Overall Description) and are then evaluated on the benchmark. That is a real circularity risk, and the paper's own ambiguity makes it impossible to audit. Appendix A.4 says annotators checked key points generated under a similarity threshold of 0.9, with a 94.7% pass rate. Section B.3 says all experimental results use a threshold of 0.8. The paper never shows the 0.8 set is covered by the human check. If it isn't, the only filter on the final set is the automated two-MLLM verification, which is exactly where bias can enter. There's also an internal inconsistency: Section 4.1 says Qwen2-VL-72B extracts key points during verification; Appendix A.3 says GPT-4 does. That matters for reproducibility.\n\nI don't think this kills the paper. The MCTS framework, the fine-tuning gains, and the human consistency data are all present, and a missing audit trail can be fixed. But it needs revision: state which threshold the human check covered, show the final set is human-verified (or re-verify), and reconcile the extractor descriptions. Without that, the benchmark rankings and the fine-tuning improvements hang on an unverified ground truth.\n\nBottom line: worth sending to serious peer review, but the referee should require the verification transparency fix. Readers in video captioning and MLLM evaluation will get value from this. I'd take it to a reading group and discuss the circularity issue.","headline":"MCTS-based video caption benchmark with a solid core and a ground-truth audit hole that needs closing before the rankings can be trusted.","tokens_in":31885,"tokens_out":2521,"would_cite":false,"duration_ms":28293,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automatic MCTS framework, AutoCaption, builds a fine-grained video captioning benchmark with 122.3 key points per video and uses the same generated data to fine-tune MLLMs.","keywords":["video captioning","multimodal large language models","Monte Carlo Tree Search","benchmark construction","key point verification","synthetic training data","fine-grained evaluation","MLLM fine-tuning"],"falsifier":"Take a random sample of the final key points produced at the official 0.8 similarity threshold, have independent human annotators judge each one against the original video, and compare the pass rate with the reported 94.7% (measured at the stricter 0.9 threshold); a large drop, or a systematic disagreement between the two verifier models, would show that the ground truth and the model rankings built on it are not anchored to reality.","tokens_in":30854,"feed_emoji":"🎬","tokens_out":8705,"duration_ms":79211,"temperature":0.7,"pith_summary":"This paper argues that a search-based automatic pipeline can replace manual annotation in building video captioning benchmarks, and that the same pipeline can generate training data that improves the models being evaluated. It introduces AutoCaption, which uses Monte Carlo Tree Search to iteratively explore six kinds of video description and verify each new detail before keeping it. The resulting benchmark, MCTS-VCB, covers 1,765 videos with an average of 122.3 key points each, split into five content dimensions. Across 21 open and closed multimodal models, Gemini-1.5-Pro scores highest, and fine-tuning InternVL2.5-8B on AutoCaption-generated data improves its F1 score by 25.0% on MCTS-VCB and 16.3% on DREAM-1K. If the framework holds up, costly manual key-point annotation can be replaced by a cheap automatic process that is also a source of training signal.","feed_headline":"MCTS-built captions lift video model scores 25 percent","feed_subtitle":"Auto-generated key points replace manual annotation and improve InternVL2.5-8B on both MCTS-VCB and DREAM-1K.","key_machinery":"The load-bearing object is the MCTS search tree over six description actions, with each node scored by the state value $Q(s,a)=0.5^{1-MC(s)}\\cdot 0.5^{SM(s)}$, where $MC(s)$ is the fraction of the node's atomic key points that pass verification by two multimodal models answering yes/no questions, and $SM(s)$ is the average similarity of the node to earlier nodes on its path. High correctness and low redundancy give a node a high value, so the search keeps expanding into new video details. Post-processing filters, deduplicates at a similarity threshold of 0.8, paraphrases, and human-checks the surviving key points to form the final benchmark.","core_discovery":"The central claim is that Monte Carlo Tree Search, a planning method from game AI, can be repurposed to enumerate the many details of a video that a good caption should mention. AutoCaption treats captioning as a search tree in which six actions (overall, detail, temporal, spatial, background, and camera-movement description) generate candidate sentences, each sentence is decomposed into atomic key points, and each key point must pass a two-model yes/no verification before it contributes to the node's score. A similarity penalty suppresses repetition, and backpropagation drives the search toward unexplored details. The verified key points become a benchmark that ranks 21 models, and the same search output, converted into thought-process-guided captions, serves as fine-tuning data that lifts a mid-size model on both the new benchmark and the existing DREAM-1K benchmark.","pith_inferences":["The same tree-search coverage mechanism could transfer to other structured perception tasks where detail coverage is the bottleneck, such as dense image captioning, spatial grounding, or long-video question answering.","Because the reward signal comes from two MLLM verifiers, the benchmark's granularity is capped by what those verifiers can perceive; a detail that both verifiers systematically miss would be excluded from the ground truth.","A natural next experiment is a self-improvement loop: fine-tune the small generator on AutoCaption data, rerun the search with the improved generator, and measure whether key points become denser and more accurate per iteration.","The reported human verification rate of 94.7% was collected at a stricter 0.9 similarity threshold while all evaluation results use 0.8, so an independent check of the 0.8-threshold key point set would clarify whether the final ground truth is as clean as the human-checked subset."],"forward_implications":["Manual key-point annotation can be replaced by an automatic MCTS search, lowering the cost of building fine-grained captioning benchmarks.","MCTS-VCB separates 21 MLLMs by five key-point dimensions and ten video genres, exposing category-specific weaknesses, with Gemini-1.5-Pro leading at an overall F1 of 71.2.","AutoCaption-generated data, without human annotation, improves InternVL2.5-8B by 25.0% on MCTS-VCB and 16.3% on DREAM-1K.","The fine-tuned model also improves on out-of-domain benchmarks such as Video-MME and MMBench-Video, suggesting the synthetic captions teach transferable video description skill.","On samples of MCTS-VCB, the benchmark's scoring correlates with human judgment more closely than existing metrics such as CLAIR, AutoDQ, BertScore, and ROUGE."],"supporting_citations":[{"why":"supplies the DREAM-1K benchmark and the key-point precision/recall/F1 evaluation protocol that MCTS-VCB adopts and extends.","marker":"(Wang et al., 2024a)"},{"why":"defines the four MCTS phases (selection, expansion, simulation, backpropagation) that AutoCaption adapts to caption generation.","marker":"(Browne et al., 2012)"},{"why":"gives the PUCT exploration term used to select the next node in the search tree.","marker":"(Rosin, 2011)"},{"why":"provides the state-value formula and the practice of using MCTS for automated data construction.","marker":"(Luo et al., 2024)"},{"why":"supports using a small MLLM as the generator that digs out detailed information inside the search.","marker":"(Qi et al., 2024)"},{"why":"is the InternVL2.5-8B model family that the paper fine-tunes on AutoCaption-generated data.","marker":"(Chen et al., 2024b)"},{"why":"is Qwen2-VL, the open generator used in the MCTS iteration and one of the two key point verifiers.","marker":"(Wang et al., 2024d)"},{"why":"is GPT-4o, used as an action executor in MCTS and as the other key point verifier.","marker":"(OpenAI, 2024)"},{"why":"is Gemini-1.5-Pro, used for initial overall descriptions and the highest-scoring model on MCTS-VCB.","marker":"(Team et al., 2024)"}],"fun_headline_variants":["Monte Carlo search writes better video captions","Game AI trick boosts video captioning benchmarks","Tree search auto-builds video caption tests and training data","MCTS turns video details into captions, lifts models 25%","AutoCaption: MCTS-crafted captions improve video AI by 25%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework rests on the assumption that two multimodal models' yes/no answers about whether a generated detail actually appears in the video are reliable enough to serve as ground truth, and that the small human-checked subset is representative of the final key point set.","fun_headline_variants_meta":{"raw":{"variants":["Monte Carlo search writes better video captions","Game AI trick boosts video captioning benchmarks","Tree search auto-builds video caption tests and training data","MCTS turns video details into captions, lifts models 25%","AutoCaption: MCTS-crafted captions improve video AI by 25%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1343,"prompt_tokens":1012,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":628,"tokens_out":331,"duration_ms":4053,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:38:40.614982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the final key points produced at the official 0.8 similarity threshold, have independent human annotators judge each one against the original video, and compare the pass rate with the reported 94.7% (measured at the stricter 0.9 threshold); a large drop, or a systematic disagreement between the two verifier models, would show that the ground truth and the model rankings built on it are not anchored to reality.","supporting_citations":[],"review_version":1}