{"id":"116127ba-7a4d-4507-9fed-9b2f2319ecd4","arxiv_id":"2608.04587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An automated evolution framework improves video-question-answering agents distribution-by-distribution, raising macro accuracy from 38.44% to 51.47% on a new eight-domain benchmark.","lead":"MetaVideoAgent automatically redesigns video-processing agents for each new video domain, using question-answer feedback on a small training set. It reports accuracy gains over fixed video agents on eight video benchmark domains, suggesting that video agents can adapt themselves rather than being hand-designed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Held-out gains hinge on question-weighted vs. macro reporting and single-run volatility; the largest danger is that the evolved-agent advantage is not distinguishable from fixed baselines once per-distribution variance is included.","rationale":"The reader's weakest_assumption focused on diagnostic fidelity and the Gold-Path attribution, which is an important internal-mechanism concern but is largely supported by the paper's own ablations (blind evolution collapses). My stress-test pass identifies a different, more load-bearing epistemic concern: the headline quantitative claims rest on single runs with no error bars, on small held-out splits, and on the paper's own reported volatility (e.g., Course Lecture volatility of 15.52 pp2 full, 131.43 pp2 DVD-initialized, and 293.24 pp2 blind). The paper is honest about limitations and the design is careful, but the central comparison to fixed baselines is not yet statistically robust. This does not overturn the paper; the most appropriate verdict remains CONDITIONAL, with the condition being the release of code/data and repeated-run variance estimates. I agree with the reader that the paper is credible but not independently reproducible, and I partially disagree on which concern is weakest: I would prioritize the single-run/no-error-bar issue over diagnostic attribution, because even perfect diagnosis cannot rescue a claim that is within noise of the baselines.","tokens_in":23636,"tokens_out":1645,"duration_ms":16895,"concrete_test":"Run MVA and each fixed video-agent baseline at least 5 times per distribution with the same evolution budget, using the released code/manifests once available, and report per-distribution mean and 95% CI. Then compute a paired or bootstrap test on the held-out question sets comparing MVA's final checkpoint against the best fixed baseline per distribution, and against the macro-average. If the average MVA advantage over the best fixed baseline falls within the CI of zero (or overlaps substantially), the 'exceeds every fixed video agent' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim is that MVA reaches 51.47% macro accuracy, 12.49 points above Gemini 3.1 Pro, and exceeds every fixed video agent. The most load-bearing concern is the statistical fragility of this comparison: there are no error bars, no repeated runs, and the V A-EvoBench held-out splits contain only 71–135 questions per distribution (867 total). The paper itself reports high volatility in Course Lecture ablations (e.g., DVD-initialized run has 131.43 pp2 volatility and blind evolution has 293.24 pp2), and the final per-distribution numbers include a 57.28→40.34 reversal in Dramatic Narrative across iterations. A single run's final checkpoint can be dominated by noise: for a 119-question split, a difference of a few questions is several points. The 'exceeds every fixed video agent by 6.39 points' claim depends on per-distribution comparison, not on a paired test across the same questions. If one recomputes per-question weighted accuracy (446/867 = 51.44%) vs. macro (51.47%), the two agree only because the per-distribution denominators are similar; but the more serious issue is that the held-out gains on several distributions (e.g., Interview 47.41 vs. DVD 45.19, or Dramatic Narrative 40.34 vs. Symphony 35.29) are within plausible run-to-run variation given the small split sizes. The paper's internal evidence (evolution-split trajectories in Fig. 6, with a wide 95% confidence band) indicates that the evolution process itself is highly variable. Without repeated independent runs, the central claim that MVA 'improves every distribution and exceeds every fixed video agent' is not established at the level of statistical significance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MetaVideoAgent proposes an automated framework that evolves a long-form-video understanding agent for a target distribution. The evolution loop iterates distribution-aware initialization (sparse frame profiling), Teacher review of failed Student trajectories (using ground-truth answers and annotated evidence intervals on an evolution split), cross-trajectory Diagnosis, and responsibility-constrained code modification over a fixed five-module agent representation (structuring, localization, perception, working memory, reasoning). Candidate acceptance is gated by engineering checks, minimal-validation probes, and full evolution-split comparison, while held-out outcomes for archived checkpoints are recorded only by an external evaluator after a fixed K-iteration budget and never enter evolution decisions. The authors also introduce VA-EvoBench, eight author-curated distributions derived from CG-Bench with video-disjoint evolution and held-out splits (356 and 867 questions, respectively). The headline results are a macro-average improvement from 38.44% to 51.47% over four iterations, 12.49 points above the strongest direct MLLM (Gemini 3.1 Pro) and 6.39 points above the best fixed video agent (WorldMM), with the lowest per-question token and frame counts among the compared video agents, at an average evolution cost of 3.54M tokens per distribution.","tokens_in":23944,"tokens_out":11840,"duration_ms":125310,"significance":"The paper's protocol discipline is a genuine strength: held-out outcomes are withheld from diagnosis, promotion, stopping, and rollback; the final agent is frozen after fixed iterations; and Appendix G lays out the annotation-access rules. The appendix pseudocode (Algorithms 1-6), the worked evolution cycles in Sections B.6 and B.7, and the planned code and data release support reproducibility. If the effect is real, the 8-of-8 improvement over the synthesized initial agents is evidence that distribution-level diagnosis can steer code-level agent evolution, and the low deployment cost (74.47K tokens/question versus 102-323K for fixed video agents) is practically meaningful. The main risk is statistical: every headline number is a single run per distribution on splits of 71-135 questions, with non-monotonic trajectories and very high volatility in the ablations (Table 2: 293.24 and 131.43 pp2), so the magnitude of the advantage over direct MLLMs and fixed agents is not yet separated from run-to-run noise. The improvement-over-initialization claim is on firmer ground than the superiority-over-fixed-agents claim.","major_comments":[{"comment":"The central accuracy figures rest on one run per distribution, and the held-out splits (Table 7) contain only 71-135 questions each (867 total). For a split of 119 questions, a move of five to six questions is several accuracy points, so the reported 6.39-point margin over WorldMM (51.47 vs 45.08) and the 12.49-point margin over Gemini 3.1 Pro are not separated from run-to-run noise by the evidence presented. Moreover, MVA does not beat fixed agents on every distribution: on Product Presentation it reaches 56.31 vs WorldMM 59.22 and Symphony 57.28, and on Dramatic Narrative 40.34 vs DVD 47.06 and WorldMM 45.38. I ask the authors to report Wilson intervals for each per-distribution accuracy, a paired comparison (McNemar or bootstrap) against at least the strongest fixed agent on the common question set, and at least two or three repeated evolution runs for one or two distributions to bound seed variance; the Course Lecture setting is the natural place since the ablation infrastructure already exists.","section":"Section 6.2 and Table 1"},{"comment":"Table 9 shows strongly non-monotonic single-run trajectories (notably Dramatic Narrative 37.82-39.50-38.66-46.22-40.34, and Gameplay 31.76-38.82-29.41-27.06-47.06), and Figure 4 reports a best-so-far held-out mean of 52.21% (95% CI 48.63-55.78) versus the 51.47% final macro accuracy. Because the reported CI is across the eight distributions rather than across repeated runs of one distribution, it cannot tell us whether the final checkpoint differs meaningfully from earlier checkpoints or from the best archived checkpoint. Please re-derive the conclusion using a measure that is robust to this, such as area under the cumulative-budget curve with a cross-run variance estimate, and report the final-checkpoint accuracy as a secondary measure.","section":"Table 9 and Figure 4"},{"comment":"The fixed-agent comparison is confounded by control-model choice. Appendix C assigns Qwen 3.7 Max as the control LLM for all four fixed video agents, while the MVA Student reasoning module uses GLM-5.2, and no direct MLLM baseline in Table 1 uses GLM-5.2. The 6.39-point advantage over fixed video agents therefore mixes agent-design differences with reasoning-model differences. To isolate the contribution of evolution, either run the fixed agents with GLM-5.2 as their planner, or run MVA with Qwen 3.7 Max as the Student, or add a GLM-5.2 direct MLLM row to Table 1. Without one of these controls, the comparison supports that evolution helps, but not the stated margin over fixed designs.","section":"Appendix C and Table 4"},{"comment":"The framework's benefit over blind evolution depends on the accuracy of Teacher Gold Paths and module attribution, and the blind ablation (Table 2, 293.24 pp2 volatility) shows the gain collapses when diagnosis is removed. Yet the paper provides no evidence on attribution accuracy: no human evaluation of a sample of Teacher reviews or diagnosis contracts, and no sensitivity analysis. Please add either a human-rated sample of diagnosis contracts (for example, 50-100 reviews on one distribution, with agreement rates on the attributed module) or a perturbation experiment that corrupts a fraction of module attributions and reports held-out accuracy as a function of corruption. Appendix J's concession that an incorrect causal attribution can yield an unhelpful diagnosis contract is exactly the point that needs empirical bounding.","section":"Sections 4.3-4.4 and Appendix J"}],"minor_comments":[{"comment":"The pseudocode and the prose disagree about the probe gate: Algorithm 6 rejects a candidate when Gprobe_k = 0, while Section B.5 states that the probe stage has no separate failure exit and an unimproved result remains actionable feedback for Codegen until PROCEED is reached. Please align the pseudocode with the intended control flow.","section":"Algorithm 6 and Section B.5"},{"comment":"Table 1 would be easier to interpret if the number of answer options and the chance-level baseline were stated, since CG-Bench multiple-choice questions typically have a fixed option count and all comparisons are made against that floor.","section":"Table 1"},{"comment":"The sentence 'It improves all eight distributions and exceeds every fixed video agent' conflates two different claims: improvement over the per-distribution initial agents (supported by Table 8) versus superiority on the macro average (not supported on Product Presentation and Dramatic Narrative, where fixed baselines are numerically higher). Please separate the two claims and, when reporting resource use, state explicitly that the fewest-token and fewest-frame comparisons are among the compared video agents, not the direct MLLMs.","section":"Section 6.2"},{"comment":"The 95% confidence interval in Figure 4 is a between-distribution interval for the mean of eight runs; the caption or text should clarify that it does not quantify run-to-run variance for a single distribution, so that readers do not over-read it as evidence of stability.","section":"Figure 4"},{"comment":"References [31] and [32] appear to describe the same Deep Video Discovery paper listed twice with different years and venues; please deduplicate and consolidate the citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"Two points for the editor's consideration. First, the reference list contains a large fraction of papers from the authors' own group, several of which are cited in the framing discussion (e.g., refs. 4-6, 9, 10, 13, 17-19, 23); this pattern weakens the related-work framing and should be trimmed toward external grounding. Second, VA-EvoBench is an author-curated regrouping of CG-Bench with manual distribution assignment (Appendix D.1); the benchmark's validity rests on curator judgment, and an inter-annotator agreement measure or a released assignment-rationale document would strengthen the claim that the eight distributions are coherent and distinct. The requested statistical additions in Major Comments 1-4 are feasible within the manuscript's scope, so I view this as a revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper deserves a real look. It carries the ADAS/AFlow line of automated agent design from text into long-form video, and does it with a genuinely careful held-out discipline: checkpoints archived by an external evaluator, held-out outcomes never touching evolution decisions, final agent frozen after a fixed number of iterations. The five-module abstraction (structuring, localization, perception, memory, reasoning) is reasonable, and the Teacher Gold Path plus minimal-validation tasks are concrete mechanisms that actually do something. VA-EvoBench, even if author-curated, is a useful common testbed: eight distributions with separate evolution and held-out splits. It will be a reference even if the method's exact gains are later revised.\n\nThe soft spots are real but mostly addressable. The biggest is statistical: every headline number comes from a single run per distribution, on held-out splits of 71–135 questions. The paper's own Course Lecture ablation shows volatility of 15–293 pp2 across settings, and the final per-distribution margins are often only a few points. For a 71-question split, two or three questions either way is several points. So \"exceeds every fixed video agent by 6.39 points\" is not established at any significance level. That is a weakness, not a dealbreaker, because the qualitative pattern—evolution improves from 38.44 to 51.47 macro average and improves on all eight distributions—is consistent even if individual margins are noisy. Second, reliance on closed-source APIs (Qwen, GLM, Codex) and the author-curated benchmark means independent reproduction requires the promised release of code and manifests; without that, the numbers are not independently checkable. Appendix J is honest about attribution fidelity and over-specialization, and the blind-evolution ablation (293 pp2 volatility) actually strengthens the case that the diagnosis machinery matters. The citation pattern is straightforward; the related work covers the right prior art, and the self-citations are background, not load-bearing.\n\nBottom line: the central claim—automated video-agent evolution can adapt to a target distribution—is credible and directionally well-supported, but the exact magnitude is not. The paper deserves peer review, with the expectation that the authors add multi-run variance, per-distribution error bars, and release code and data. I would want those before relying on the 6.39-point claim, but the method and benchmark are worth engaging now. Yes, send to review.","headline":"A credible first cut at auto-evolving video agents with an honest held-out protocol, but the headline margins rest on single runs without variance bars.","tokens_in":24557,"tokens_out":2463,"would_cite":true,"duration_ms":27369,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically evolving a video agent for its target domain raises macro accuracy from 38.44% to 51.47% across eight video domains, beating every fixed video agent and direct LLM with fewer tokens and frames per question.","keywords":["automated agent evolution","long-form video understanding","video agent","distribution-adaptive design","evidence-grounded diagnosis","modular agent representation","minimal validation task","video question answering"],"falsifier":"Run the same four-iteration evolution on VA-EvoBench with the Diagnosis Agent's module attribution scrambled — for example, always blaming the perception module regardless of the Teacher's review — and record held-out macro accuracy. If accuracy still climbs roughly 13 points, diagnostic fidelity is not the cause of the reported gains; if it collapses toward the blind-evolution trajectory, the evidence-grounded diagnosis is confirmed as the load-bearing mechanism.","tokens_in":23422,"feed_emoji":"🎬","tokens_out":19328,"duration_ms":183505,"temperature":0.7,"pith_summary":"Long-form video understanding requires locating sparse, question-relevant evidence inside long multimodal recordings, and no single fixed agent design suits every video domain: interviews hide the answer in speech, stage performances in brief visual events, and films in relations across distant scenes. MetaVideoAgent claims that a video agent can instead be evolved automatically for a target distribution, using a small set of labeled examples to profile the domain, aggregate recurring failures across samples, attribute them to responsible modules, and apply code-level updates there. The paper reports that four evolution iterations per distribution improve macro accuracy from 38.44% to 51.47% on its new VA-EvoBench, beating the strongest fixed video agent by 6.39 percentage points while using the fewest tokens and frames per question among all video agents compared. If the results hold, video-agent design becomes an adaptation loop driven by distribution-level feedback rather than a one-time hand-engineering effort.","feed_headline":"Video agents that evolve themselves gain 13 points on long-video QA","feed_subtitle":"Eight video domains each get a distinct evolved agent, with fewer tokens and frames than the fixed rivals it beats.","key_machinery":"The load-bearing object is the Gold Path: a minimal sufficient evidence chain that the Teacher Video Agent reconstructs from the question, the ground-truth answer, the annotated evidence interval, and the raw media before inspecting the Student's trajectory. It specifies which facts must hold, which channels establish them, and how they entail the answer, and it lets the Teacher find the earliest Student step that can no longer support the correct solution. Around it sits the modular agent representation $A = (S, L, P, W, R)$ — video structuring, evidence localization, perception, working memory, and reasoning — which constrains each code edit to the primary responsible module and only the dependencies needed to preserve interfaces and end-to-end execution. Completing the mechanism are minimal validation tasks, which compress localized failures into independently executable probes so a candidate can be tested cheaply before the full evolution-split comparison.","core_discovery":"The paper's central claim is that code-level evolution of video agents is feasible and beats hand-designed alternatives. Starting from a distribution profile built from five sparsely sampled frames per video plus the associated query set, an initial agent is synthesized; four iterations of execution, review, diagnosis, and constrained code modification then produce an agent that improves all eight distributions of VA-EvoBench and surpasses every fixed video agent and direct multimodal LLM in the comparison. The demonstration rests on two mechanisms the paper introduces: a five-module representation of a video agent, namely video structuring, evidence localization, perception, working memory, and reasoning, which lets each code update be scoped to the primary responsible module plus its necessary dependencies; and an evidence-grounded Gold Path, reconstructed by a Teacher Video Agent from the ground-truth answer and annotated evidence intervals, which audits the Student's trajectory and locates the earliest step that can no longer support the correct answer. The paper's ablations attribute the stability of the process to this diagnostic stack: without review, diagnosis, and targeted research, evolution becomes highly volatile (293.24 pp² update variance versus 15.52 for the full method), while initializing from an existing fixed agent (DVD) improves it further, to 63.03% on Course Lecture.","pith_inferences":["If the paper is right, the practical bottleneck for adapting an agent to a new video domain shifts from pipeline engineering to producing labeled video-QA examples with annotated evidence intervals; the natural next step the authors do not claim is replacing expert annotations with automatically generated Gold Paths.","The reusable behaviors the paper documents — event ordering, position-relation perception, cross-view subject tracking — suggest that recurring evidence patterns could be captured as portable module upgrades and shared across distributions, a transfer the paper explicitly does not establish.","The reported per-question token and frame costs are deployment costs only; evolution itself consumes 3.54M tokens per distribution on average, so a practitioner comparing evolved against fixed agents should amortize that one-time evolution cost into the per-question budget."],"forward_implications":["Evolved agents would become the practical ceiling for video QA on a target domain: MetaVideoAgent's final agents exceed the best fixed design by 6.39 points while using fewer tokens (74.47K vs 102.26K per question) and frames (29.11 vs 42.77) than any fixed video agent compared.","The gains are not confined to easy distributions: evolution improves every initial agent, including the weakest start (Stage Performance, from 23.30% to 57.28%).","Evolution can also upgrade a hand-built agent instead of starting from scratch: initializing from an existing fixed design (DVD) raises Course Lecture accuracy by a further 3.37 points, to 63.03%.","Learned behaviors transfer to unseen questions within a distribution: the evolved capabilities shown — event ordering, position-relation perception, cross-view subject tracking — are demonstrated on held-out questions that played no role in evolution.","Most of the gain arrives within the fixed budget: the mean best-so-far held-out accuracy across the eight runs reaches 52.21% (95% CI 48.63–55.78) at full budget, with evolution costing on average 3.54M tokens per distribution."],"supporting_citations":[{"why":"CG-Bench supplies all videos, questions, answers, and annotated evidence intervals from which VA-EvoBench's eight distributions are built; without it the evolution protocol has no data.","marker":"[2]"},{"why":"ADAS contributes the evolution objective E(A_in, D_tr) and the execute-diagnose-evolve loop that MetaVideoAgent extends from text tasks to video agents.","marker":"[8]"},{"why":"Deep Video Discovery is the fixed video-agent baseline whose open implementation anchors the Table 1 comparison and the from-DVD initialization ablation (63.03% on Course Lecture).","marker":"[32]"},{"why":"M3-Agent is one of the four fixed video-agent baselines whose functional mapping grounds the five-module abstraction.","marker":"[15]"},{"why":"Symphony is the fixed multi-agent video baseline whose high per-question cost (322.94K tokens, 830.18 frames) motivates the efficiency comparison.","marker":"[24]"},{"why":"WorldMM is the strongest fixed video agent in the comparison (45.08% average) that MetaVideoAgent surpasses by 6.39 points.","marker":"[25]"},{"why":"OpenAI Codex serves as the code-level Evolution Agent that implements the modular updates generated from the diagnosis contract.","marker":"[16]"}],"fun_headline_variants":["Video agents evolve themselves to beat fixed designs on 8 video domains","Self-evolving video agents gain 13 points, use fewer tokens and frames","Automated agent evolution for long-video QA lifts accuracy 13 points","Video agents that evolve code outperform fixed rivals on long-video tasks","MetaVideoAgent evolves per-domain agents, surpassing all fixed baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on one premise: the Teacher's Gold Path and the Diagnosis Agent's module attributions are accurate enough that editing the blamed module actually repairs the failure — the paper concedes in Appendix J that an incorrect causal attribution yields an unhelpful diagnosis contract, and its own blind-evolution ablation (volatility 293.24 pp² against 15.52 for the full method) shows the gain depends on this fidelity.","fun_headline_variants_meta":{"raw":{"variants":["Video agents evolve themselves to beat fixed designs on 8 video domains","Self-evolving video agents gain 13 points, use fewer tokens and frames","Automated agent evolution for long-video QA lifts accuracy 13 points","Video agents that evolve code outperform fixed rivals on long-video tasks","MetaVideoAgent evolves per-domain agents, surpassing all fixed baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4067,"prompt_tokens":1086,"completion_tokens":2981,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":2888}},"tokens_in":702,"tokens_out":2981,"duration_ms":23079,"temperature":1.0,"reasoning_tokens":2888,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:21:39.929693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four-iteration evolution on VA-EvoBench with the Diagnosis Agent's module attribution scrambled — for example, always blaming the perception module regardless of the Teacher's review — and record held-out macro accuracy. If accuracy still climbs roughly 13 points, diagnostic fidelity is not the cause of the reported gains; if it collapses toward the blind-evolution trajectory, the evidence-grounded diagnosis is confirmed as the load-bearing mechanism.","supporting_citations":[{"cited_title":"Cg-bench: Clue-grounded question answering benchmark for long video understanding, 2024","cited_arxiv_id":null,"evidence_quote":"CG-Bench supplies all videos, questions, answers, and annotated evidence intervals from which VA-EvoBench's eight distributions are built; without it the evolution protocol has no data."},{"cited_title":"Automated design of agentic systems","cited_arxiv_id":null,"evidence_quote":"ADAS contributes the evolution objective E(A_in, D_tr) and the execute-diagnose-evolve loop that MetaVideoAgent extends from text tasks to video agents."},{"cited_title":"Deep video discov- ery: Agentic search with tool use for long-form video understanding, 2025","cited_arxiv_id":null,"evidence_quote":"Deep Video Discovery is the fixed video-agent baseline whose open implementation anchors the Table 1 comparison and the from-DVD initialization ablation (63.03% on Course Lecture)."},{"cited_title":"Seeing, listen- ing, remembering, and reasoning: A multimodal agent with long-term memory","cited_arxiv_id":null,"evidence_quote":"M3-Agent is one of the four fixed video-agent baselines whose functional mapping grounds the five-module abstraction."},{"cited_title":"Symphony: A cognitively-inspired multi-agent system for long-video understanding","cited_arxiv_id":null,"evidence_quote":"Symphony is the fixed multi-agent video baseline whose high per-question cost (322.94K tokens, 830.18 frames) motivates the efficiency comparison."},{"cited_title":"Worldmm: Dynamic multimodal memory agent for long video reasoning, 2025","cited_arxiv_id":null,"evidence_quote":"WorldMM is the strongest fixed video agent in the comparison (45.08% average) that MetaVideoAgent surpasses by 6.39 points."},{"cited_title":"Openai codex","cited_arxiv_id":null,"evidence_quote":"OpenAI Codex serves as the code-level Evolution Agent that implements the modular updates generated from the diagnosis contract."}],"review_version":1}