{"id":"dbb43c2d-62d8-4a8a-b020-581d06d51c9a","arxiv_id":"2505.16784","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free ensemble of commercial VLMs with prompt and chain-of-thought engineering reaches 79% on EgoSchema, ranking 2nd in the CVPR 2025 challenge, but the ensemble weights are fit to the test labels.","lead":"This paper describes the runner-up solution for the Ego4D EgoSchema video question answering challenge at CVPR 2025, using prompt engineering and an ensemble of commercial multimodal models to reach 79% accuracy on the benchmark. The report may interest practitioners because it shows a training-free pipeline can beat earlier fine-tuned systems, though the reported ensemble accuracy is weakened by the use of test labels to set voting weights.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 79% headline is an in-sample fitted statistic: ensemble weights and mode selection are computed from ground-truth labels on the same test set used for scoring, so the reported accuracy is not a valid out-of-sample estimate.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing flaw: the ensemble weights in Equation 1 and the mode-selection similarity in Figure 6 are both computed from ground-truth labels on the same 5000-sample test set used to report the final accuracy. The paper openly states this in Sections 6.5 and Table 3, and Equation 3 uses those weights to compute the final vote. There is no indication of any held-out procedure, cross-validation, or separate development set for the ensemble. Reporting 79% under these conditions is circular, because the ensemble has been optimized to the test answers themselves. This is not merely a stylistic concern about reporting; it means the central numerical claim cannot be interpreted as a prediction of performance on unseen data.\n\nA separate but reinforcing issue is the claimed single-model superiority over iLearn. Mode 1 achieves 75.9% test accuracy and is described as outperforming the previous 75.0% SOTA, but the gap is small, no uncertainty is given, and the mode was selected after observing test accuracy across several configurations. The validation ablations are genuinely useful and partially mitigate concerns about prompt and CoT engineering, but they do not cover the ensemble construction. Because the headline result is an artifact of using test labels, the verdict of REJECT is appropriate: the paper's main conclusion is not supported as a valid out-of-sample finding, even though the prompt-design insights may carry practical value.","tokens_in":8291,"tokens_out":2042,"duration_ms":18845,"concrete_test":"Re-run the final ensemble (Equation 3) using weights and similarity scores computed only from the 500-sample validation set rather than from the 5000-sample test set: set w_k and sim(a,b) from validation predictions and ground-truth labels, then evaluate the resulting weighted vote on the test set. If the test accuracy falls materially below 79% (e.g., below 77%), the reported number depends on in-sample test labels and the central claim is not generalizable.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that the complementary ensemble reaches 79% accuracy on the EgoSchema test set, ranking 2nd in the CVPR 2025 challenge. This claim depends on the ensemble weights being a legitimate, generalizable combination of model outputs. The paper's own description violates that condition. In Section 6.5, Equation 1 defines each mode's weight as w = sum_i I(y_i = yhat_i)/n, where y_i is the ground-truth label; Table 3 then lists 'Accuracy (%) of different modes in 5000 test dataset' and Figure 6 shows the similarity matrix computed on the same 5000 test samples. The text states that seven modes are selected because they have 'high accuracy and low similarity' — both criteria evaluated on test labels. The final weighted vote in Equation 3 therefore uses test-derived weights and test-derived mode selection to report accuracy on that same test set. This is circular: the 79% is an in-sample fitted statistic, not an unbiased estimate of how the method would perform on unseen data.\n\nA secondary but related weakness is the abstract's claim that a single VLM already outperforms the previous SOTA. The cited comparison is 75.9% (Mode 1, Table 3) versus 75.0% (iLearn). That 0.9-point gap is within likely noise, the paper reports no error bars or significance test, and Mode 1 was itself observed after inspecting test-set accuracy for multiple candidate modes. The validation-set ablations (Section 6.2, Table 2) are more credible because they are out-of-sample relative to the final ensemble, but they do not establish the headline 79%.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the runner-up solution for the Ego4D EgoSchema Challenge at CVPR 2025. It proposes a 1-stage prompting pipeline for video-language models, systematically compares prompt styles and chain-of-thought output structures, and combines seven complementary modes with a weighted ensemble. The headline claim is that this ensemble reaches 79% accuracy on the 5000-sample EgoSchema test set, improving on the previous SOTA method iLearn at 75%, and that even a single mode (75.9%) already surpasses that SOTA. The supplementary material provides ablation studies on a 500-sample validation set and detailed equations for the ensemble weights and mode similarity.","tokens_in":8661,"tokens_out":2605,"duration_ms":19189,"significance":"If the reported result were a valid out-of-sample estimate, this would be a useful practical demonstration that training-free, 1-stage prompting and ensembling of accessible VLMs can compete at the top of a long-form video QA leaderboard. The paper has concrete strengths: it makes code available, reports a systematic ablation of CoT components, and is transparent about the configuration choices and their evolution. However, the central 79% figure is computed through a circular evaluation procedure: the ensemble weights and mode selection are derived from ground-truth labels on the same test set that is used to report the final accuracy. This invalidates the main quantitative claim as an unbiased performance estimate, and it also undermines the secondary claim that a single mode beats the prior SOTA. The validation-set ablations are less affected by this issue, but they are not the paper's headline result.","major_comments":[{"comment":"The reported 79% accuracy is an in-sample fitted statistic. Eq. (1) defines each mode's weight w as the fraction of samples for which that mode matches the ground-truth label y_i, and Table 3 explicitly reports \"Accuracy (%) of different modes in 5000 test dataset.\" Figure 6 computes the similarity matrix on the same 5000 test samples, and Section 6.5 says the seven modes are selected because they have \"high accuracy and low similarity.\" The final weighted vote (Eq. (3)) then uses these test-derived weights and test-derived mode selection to report accuracy on that same test set. This is circular: the same y_i appear in the weight computation and in the final evaluation. The claim in Section 3 that the method \"outperforms almost all previously proposed methods (75%→79%)\" is therefore not supported as an unbiased, generalizable result. The authors should recompute weights and select modes using only the validation split, or clearly re-frame the reported numbers as post-hoc fits to the test set rather than predictive performance.","section":"Section 6.5, Eq. (1), Table 3, Figure 6"},{"comment":"The abstract's claim that \"directly utilizing an individual multimodal model already outperforms the previous SOTA\" rests on a 0.9-point gap (Mode 1 at 75.9% versus iLearn at 75.0% in Table 1/Table 3). No confidence intervals, significance tests, or repeated-run variance are reported, and Mode 1 was itself selected after inspecting its accuracy on the test set during the ensemble construction (Section 6.5). A 0.9-point difference on 5000 samples, with no uncertainty quantification, is not sufficient to support a claim of superiority, especially when the mode was chosen post hoc on the same test set.","section":"Abstract; Section 3; Table 1"},{"comment":"The validation-set ablations are more credible because they are out-of-sample with respect to the 500-sample validation split, but the paper itself notes \"a stark difference between the trends observed in the validation set and the test set, indicating a significant data bias.\" This means the validation set may not reliably select the best CoT configuration, and the later test-set-based selection of the seven modes (Section 6.5) is exactly the kind of multiple-comparison procedure that inflates apparent accuracy. The authors should provide separate validation-based and test-based estimates, and should avoid drawing conclusions about the best configuration from test-set numbers alone.","section":"Section 6.2; Section 6.5"}],"minor_comments":[{"comment":"There is a typo in \"we develope a 1-stage process\" which should read \"we develop a 1-stage process.\"","section":"Section 2.2"},{"comment":"Figure 2 is visually dense and the legend for the diagonal line and green dashed line is easy to miss; the caption should define all symbols directly in the figure caption.","section":"Figure 2"},{"comment":"The text says \"The similarity between any two modes A and A\" but should say \"A and B\"; the equation is correct but the surrounding sentence is sloppy.","section":"Section 2.5"}],"recommendation":"reject","confidential_remarks":"The core evaluation is circular, and the headline 79% cannot be salvaged by a local revision; correcting it would require recomputing the ensemble with validation-derived weights and likely changing the main result. The paper may still be of interest as a workshop-style competition report, but as a journal submission the central claim is not reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competition write-up, and its central number does not survive contact with its own equations. The 79% accuracy on EgoSchema comes from a weighted vote in which the weights are per-mode accuracies computed from ground-truth labels on the same 5000-sample test set (Eq. 1, Table 3), and the seven ensembled modes are selected because they have high accuracy and low similarity on that same test set (Figure 6). That makes the headline an in-sample fitted statistic, not an unbiased estimate of how the method would do on unseen data. The abstract's single-model claim (75.9% vs iLearn's 75.0%) is also weaker than it looks: that mode was observed after inspecting test-set accuracy for multiple configurations, and the 0.9-point gap comes with no error bars or significance test.\n\nWhat is actually good here is the validation-side work. Table 2 and the supplementary prompt examples give a useful, honest picture of how CoT components, prompt styles, and the 1-stage vs 2-stage design affect accuracy on a held-out 500-sample validation set. The observation that a single end-to-end VLM prompt can beat a more elaborate 2-stage pipeline is practically useful and worth reporting. The code is released, and the paper does not hide its evolution path (Figure 2). The authors even note that validation and test trends diverge (Section 6.2), which is a red flag for anyone wanting to trust test-set-tuned numbers.\n\nThe circularity is not a minor blemish; it undermines the main performance claim. But the paper is not incoherent, and the validation insights are reproducible and useful to practitioners. If the ensemble weights had been fixed on the validation set and the test set held out for one final evaluation, the 79% could have been credible. As written, it is a competition report with a flawed evaluation protocol.\n\nWho should read this: people doing prompt engineering for long-form video QA, or anyone curious about how far training-free ensembles of commercial VLMs can go. It is not a rigorous methodological contribution. I would send it to a serious referee if the venue is a workshop or if the authors commit to fixing the evaluation with a proper validation/test split; for a top-tier conference, the central claim as presented would fail review. My own verdict: reject for the reported result, but the paper deserves referee time because the validation-side findings and the 1-stage comparison are worth capturing in the literature after a rewrite.","headline":"The headline 79% is in-sample fitted because the ensemble weights are computed from ground-truth labels on the very test set used for scoring; the genuinely useful material is the validation-side comparison of prompts, CoT, and 1-stage vs 2-stage pipelines.","tokens_in":9198,"tokens_out":3215,"would_cite":false,"duration_ms":22272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free ensemble of seven large-model “thinking modes” answers long egocentric video questions at 79% accuracy, beating the prior 75% state of the art.","keywords":["EgoSchema","video question answering","vision-language models","prompt engineering","chain-of-thought","model ensemble","egocentric video","training-free adaptation"],"falsifier":"Compute the seven mode outputs on the 5,000 test questions, hold out one half of the test set, estimate the ensemble weights and mode selection from that half only, and score the ensemble on the held-out half. If the held-out accuracy falls to or below the single-mode baseline (around 75%), the 79% figure is largely an artifact of test-set weighting. A weaker check: perturb the weight vector by small noise and see whether the final answer changes; if it does, the ensemble result is fragile.","tokens_in":7989,"feed_emoji":"🎬","tokens_out":5317,"duration_ms":42606,"temperature":0.7,"pith_summary":"This paper argues that, for long-form video question answering, a single forward pass through a modern vision-language model with carefully chosen prompts can outperform multi-stage pipelines that first caption a video and then reason over the caption. The authors show that a one-stage design, in which the model outputs captions, summary, reasoning, answer, and confidence in one JSON response, reaches 75% or better on its own. Combining seven such 'modes'—different prompt styles, chain-of-thought structures, and models—through similarity-weighted voting raises accuracy to 79%, which placed second on the EgoSchema challenge leaderboard. The practical claim is that collaboration between large models can be harvested without any additional training or expensive multi-stage orchestration.","feed_headline":"Seven-prompt AI ensemble hits 79% on long video questions","feed_subtitle":"A single vision-language model with tuned prompts already beats the previous 75% state of the art.","key_machinery":"The load-bearing mechanism is the complementary-ensemble voting rule. Each mode $k$ is assigned a weight $w_k$ equal to the fraction of samples on which it agrees with ground truth ($w = \\sum_i \\mathbb{I}(y_i = \\hat{y}_i)/n$), and each pair of modes has a similarity $\\mathrm{sim}(A,B)$ equal to their answer-overlap fraction. The ensemble score for option $c$ is $\\sum_k w_k \\sum_b \\mathrm{sim}(a,b) \\, \\mathbb{I}(\\hat{y}_k = c)$, and the answer with the highest score wins. The similarity term is meant to stop near-duplicate modes from dominating the vote. Equally important is the one-stage prompt/CoT construction: the model is asked to emit caption, summary, reason, answer, and confidence in a single JSON response, which the paper finds both simpler and more accurate than the earlier two-stage caption-then-reason pipeline.","core_discovery":"On the EgoSchema benchmark, the paper's central discovery is that the bottleneck is not video understanding capability but how the model's attention is directed. A 1-stage prompt that asks a vision-language model to produce a clip-by-clip caption, a global summary, a reason, an answer, and a confidence score in one JSON block outperforms the previous state-of-the-art 75% method, which used a separate captioning stage plus a reasoning stage. The paper then treats each prompt style and model as a distinct 'mode' of thinking and ensembles the most accurate, least redundant modes. With seven modes, ensemble weights are set to each mode's accuracy and modulated by pairwise answer similarity, and the final answer is the option receiving the highest similarity-weighted vote; this reaches 79% on the 5,000-question test set. In the paper's implementation, these weights and similarities are computed on the test set itself.","pith_inferences":["The paper's design uses the test set's ground-truth labels to choose ensemble weights and to select the seven modes, then reports accuracy on that same test set; unless those choices generalize to a held-out split, 79% is an upper bound on what a deployment would see.","A direct testable extension is to estimate weights on a randomly chosen half of the test set and score the ensemble on the other half; the gap, if any, would quantify how much of the gain is genuine complementarity versus fitting.","The paper's 'confidence' output component does not appear to enter the ensemble rule (weights are answer-accuracy, not self-reported confidence); a natural variant is to blend self-confidence with empirical accuracy to reduce reliance on labels.","The same prompt-differentiation recipe could be applied to other multiple-choice video QA benchmarks where a few hundred labeled examples are available, but the reported advantage may shrink when the label budget is too small to estimate weights reliably."],"forward_implications":["A single strong vision-language model with a well-structured one-stage prompt can replace a multi-stage caption-then-reason system for long-form video QA, cutting cost and complexity.","Differentiated prompt styles and chain-of-thought output structures produce answers that are complementary enough that voting over them beats any single mode.","Because all model calls are parallel and the ensemble is a lightweight vote, the final solution remains effectively one stage and near-zero-cost at inference.","If the 79% transfers to other benchmarks, training-free prompt-and-ensemble recipes become a practical baseline that future video-QA methods must beat.","Careful ablation shows that long, example-heavy prompts can hurt performance, so prompt design should be treated as a first-class experimental variable."],"supporting_citations":[{"why":"Supplies the 5,000-question EgoSchema benchmark and test set on which all accuracies are reported.","marker":"[9]"},{"why":"The previous state-of-the-art 75% method and the 2-stage caption-then-reason pipeline that the paper compares against and improves on.","marker":"[17]"},{"why":"A representative 2-stage approach used as the starting point in the paper's macro-design ablations.","marker":"[13]"},{"why":"An earlier prompt-and-assembly framework that motivates the training-free deployment paradigm.","marker":"[16]"},{"why":"The LaViLa video captioner used in the 2-stage variants, showing that fine-tuned captioners can contribute in that design.","marker":"[18]"},{"why":"A strong video-language model baseline whose reported 63% accuracy anchors the comparison table.","marker":"[11]"}],"fun_headline_variants":["Single tuned prompt beats 75% SOTA; seven-mode ensemble hits 79%","Prompt diversity and similarity-weighted voting lift EgoSchema QA to 79%","JSON-style prompts direct attention, lifting video QA to 79%","From 75% to 79%: prompt tuning and ensemble on EgoSchema","Seven-prompt ensemble scores 79% on EgoSchema video QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 79% rests on the assumption that choosing ensemble weights and modes by looking at the test set's correct answers, and then reporting accuracy on that same test set, gives an honest measure of performance; if that assumption is false, the number is not a generalizable result.","fun_headline_variants_meta":{"raw":{"variants":["Single tuned prompt beats 75% SOTA; seven-mode ensemble hits 79%","Prompt diversity and similarity-weighted voting lift EgoSchema QA to 79%","JSON-style prompts direct attention, lifting video QA to 79%","From 75% to 79%: prompt tuning and ensemble on EgoSchema","Seven-prompt ensemble scores 79% on EgoSchema video QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001415,"raw_usage":{"total_tokens":5699,"prompt_tokens":918,"completion_tokens":4781,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":4679}},"tokens_in":534,"tokens_out":4781,"duration_ms":24693,"temperature":1.0,"reasoning_tokens":4679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:54:51.473375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the seven mode outputs on the 5,000 test questions, hold out one half of the test set, estimate the ensemble weights and mode selection from that half only, and score the ensemble on the held-out half. If the held-out accuracy falls to or below the single-mode baseline (around 75%), the 79% figure is largely an artifact of test-set weighting. A weaker check: perturb the weight vector by small noise and see whether the final answer changes; if it does, the ensemble result is fragile.","supporting_citations":[{"cited_title":"Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the 5,000-question EgoSchema benchmark and test set on which all accuracies are reported."},{"cited_title":"Learning video representations from large lan- guage models","cited_arxiv_id":null,"evidence_quote":"The LaViLa video captioner used in the 2-stage variants, showing that fine-tuned captioners can contribute in that design."}],"review_version":1}