{"id":"68a0ddf6-99d2-4d77-b4aa-f9a8d827f348","arxiv_id":"2608.12290","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An agentic two-stage optimizer (LLM-generated question scoring plus Bayesian hyperparameter search) improves image-to-video prompt adherence, winning human preference tests up to 69% over random search.","lead":"This paper introduces a two-stage \"Agentic Self-Improvement\" pipeline that first rewrites text prompts using an AI model's own generated questions, then uses Bayesian optimization to tune random seeds and guidance scales for image-to-video generation. In human preference tests, videos from the pipeline beat random-search baselines up to 69% of the time, suggesting a practical path toward more controllable video generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Report the optimization objective values and run a null-objective control; otherwise the claimed Bayesian-search advantage over Best-of-Random lacks a demonstrated mechanism.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the automated metrics guiding the optimizer are not shown to be sensitive to the quality differences that drive human preference, and their values are never reported. I agree this is the central gap. The strongest_claim depends on the Bayesian search being more than an elaborate random sampler; if the reward signal is flat or noisy, the advantage over Best-of-Random lacks a mechanism. The paper's own admission in Appendix 7.1 that standard automated metrics are insensitive strengthens this concern rather than resolving it. The human preference results are real evidence that the overall pipeline produces preferred videos, but they do not, by themselves, show that the Bayesian search contributes beyond the hard filter and final metric-based selection. A concrete control — comparing Bayesian search against random search with identical filtering and selection — would settle whether the search itself is doing work. The existing Best-of-Random baseline is a step in this direction, but it conflates the search algorithm with the selection metric; replacing only the search component is the needed isolation. Because the reader already conditioned the verdict on these evaluation limitations, my concern does not change the verdict; it sharpens the condition. No ad hominem is intended; the critique is on the evidence chain for the search mechanism.","tokens_in":13503,"tokens_out":2231,"duration_ms":23195,"concrete_test":"Re-run the 100-generation budget on the same 100 V-Bench image-prompt pairs with the Bayesian optimizer replaced by random search, keeping the identical hard artifact filter and the same Top-1 metric selection procedures, then recompute the human preference win rates in Table 2. If the Bayesian-search win rate does not exceed the random-search-selection win rate within confidence intervals, the claimed search benefit is not established. Additionally, report the distribution of RAHF, UVQ, and VTA over all candidates generated by both methods, and plot the running best objective per iteration; if the Bayesian optimizer's best-objective curve is flat or its objective distribution overlaps the random-search distribution, the search is maximizing an uninformative signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the complete two-stage agentic pipeline, including Bayesian search over seeds and CFG scales, yields videos that human evaluators prefer over random search under the same compute budget. The end-to-end comparison against Best-of-Random (Table 2) is the right experimental isolation of the search itself, but the mechanism asserted is that the multi-objective reward — RAHF, UVQ, and VTA — provides an informative signal that guides the optimizer toward better regions of parameter space. That mechanism is never directly evidenced. The objective values observed during search are not reported anywhere, so we cannot tell whether the Bayesian optimizer's chosen candidates actually have higher RAHF/UVQ/VTA than typical random candidates. The paper's own Appendix 7.1 (Table 3) shows that V-Bench++ metrics are nearly flat across all conditions and argues these metrics lack sensitivity to the differences that drive human preference. If RAHF, UVQ, and VTA are similarly flat or dominated by noise, the Bayesian optimizer is maximizing noise, and the 42%-vs-8% win-rate advantage over Best-of-Random could instead be attributable to the hard artifact filter or to the final Top-1 selection metric being partially informative, not to the search trajectory. A further complication is that VTA is self-referential: the mLLM generates the DSG/CMQ questions, answers them, and the resulting score steers the optimizer; agreement with human labels was measured at only 82% on CMQ in a 100-item validation. This is not a fatal flaw, but it increases the burden on the paper to show the search objective is actually moving. Across all experiments, no convergence curves, no per-iteration best-objective values, and no correlation between objective score and human preference at the candidate level are provided. The load-bearing assumption is that the optimizer's reward function is informative, and this assumption is currently unsupported by direct evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 'Agentic Self-Improvement,' a two-stage framework for controlling black-box image-to-video (I2V) generation. Stage one iteratively refines the text prompt using an mLLM that generates Davidsonian Scene Graph (DSG) questions and Common Mistake Questions (CMQ), then uses its own yes/no answers as an alignment score to drive prompt rewriting. Stage two uses Bayesian optimization (via Vizier) over CFG scales and random seeds, guided by a multi-objective reward combining RAHF, UVQ, and a novel Video-Text Adherence (VTA) score derived from the DSG/CMQ tree. The paper reports a human preference study on 100 V-Bench image-prompt pairs, claiming win rates up to 69% against random search and 42% against a Best-of-Random baseline, and concludes that the closed-loop optimizer actively finds higher-quality parameter regions. The central claim is that the complete pipeline yields perceptibly better videos than unguided search under the same compute budget.","tokens_in":13803,"tokens_out":3350,"duration_ms":32620,"significance":"If the central claim holds, the framework is a useful practical contribution: it offers a modular, model-agnostic wrapper around black-box I2V APIs, and the human-preference evaluation is a step beyond purely automated benchmarks. The paper also makes a methodological point that standard automated metrics (V-Bench++) are insensitive to the quality differences humans perceive, which is worth taking seriously. However, the manuscript does not currently demonstrate the mechanism by which Bayesian search outperforms random search: the reward values observed during optimization are never reported, and the VTA objective is generated and scored by the same mLLM that the prompt optimizer is trying to satisfy. The significance of the claimed gains therefore rests on unverified assumptions about the informativeness of the reward signal.","major_comments":[{"comment":"The paper's strongest claim is that the Bayesian optimizer 'actively and effectively guides the generation process into higher-quality parameter regions,' evidenced by 42% vs. 8% win rates against Best-of-Random. This inference is not supported because the objective values (RAHF, UVQ, VTA) observed during the 10- and 100-run searches are never reported. If these objectives are nearly flat or dominated by noise—as the paper itself argues for V-Bench++ metrics in Appendix 7.1—then the reported advantage could come almost entirely from the hard artifact filter and the final Top-1 selection, not from the search trajectory. The authors should report the distribution of each reward component across BO iterations and across random candidates, and ideally run a null-objective control where BO optimizes a random or shuffled reward, to demonstrate that the search itself is responsible for the win-rate gap.","section":"Section 4.2, Table 2"},{"comment":"The VTA score is self-referential in a way that is load-bearing for the optimization claim: the same mLLM (Gemini 2.5 Pro) generates the DSG/CMQ question trees, answers them, and the prompt optimizer then rewrites prompts to maximize those answers. The human validation in Section 3.1.3 reports 87% overall agreement on 100 video-question pairs, but this does not validate that VTA rankings correlate with human preferences at the candidate-selection level, nor does it rule out the optimizer exploiting mLLM answer biases (e.g., rewording prompts to make questions easier to answer affirmatively). The authors should report per-question-type agreement, inter-annotator agreement if both annotators scored the same items, and a direct correlation between VTA scores and human preference judgments on the candidate pools used in Table 2.","section":"Section 3.2.1, VTA definition"},{"comment":"The prompt optimization loop's effectiveness is not quantitatively documented. The paper states that the loop runs for a fixed number of iterations and keeps the prompt with the highest alignment score, but it does not report alignment-score trajectories, the number of iterations used, or any example of the original versus optimized prompt. Without this evidence, it is unclear whether iterative rewriting actually improves semantic adherence or merely changes surface phrasing that happens to score higher on the mLLM's own questions. Providing before/after prompts and score trajectories is necessary to support the claim that prompt optimization is a meaningful component of the framework.","section":"Section 3.1.2 and Section 4.1"},{"comment":"The human preference study uses only two expert annotators, each evaluating 50 disjoint prompts, with no inter-annotator agreement computed. This makes it impossible to assess the reliability of the preference labels. The paper should either have both annotators score an overlapping subset, report Cohen's kappa or a similar agreement measure, or otherwise justify that a single annotator per prompt is sufficient for the reported binomial test to be meaningful. This is not fatal to the paper's central claim, but it weakens the strength of the headline 69% win rate.","section":"Section 4, Human preference study"}],"minor_comments":[{"comment":"The author line contains apparent formatting artifacts: 'AMAN TY AGI, HEMANTH BOINPALL Y ET. AL.' and 'Aman T yagi' should be cleaned up before submission.","section":"Title page"},{"comment":"The figure caption includes a local file path 'file:///Users/brhemanth/Downloads/...' that should be removed.","section":"Figure 1 caption"},{"comment":"The table formatting is garbled; the numbers appear to run together (e.g., '144739'), making raw counts and percentages unreadable. Please reformat the table with clear column separators.","section":"Table 2"},{"comment":"The sentence on statistical significance uses superscripts such as 'p<10−6' without rendered superscript formatting; clarify whether the reported p-values are two-tailed binomial tests on the pooled 100 prompts or on a per-annotator basis.","section":"Section 4.2"},{"comment":"The VTA equation uses 'S_p(i)' without defining p(i) explicitly in the text; please define the parent mapping and state whether the root mask S_p(root)=1 applies to all root nodes.","section":"Section 3.2.1"},{"comment":"Reference [41] lists 'Kai Zo' as an author; this appears to be a truncation and should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising practical framing and the human-preference comparison against Best-of-Random is the right experimental design, but the missing objective-value reporting is a substantive gap that prevents verification of the claimed search advantage. I would advise the editor to request a revision that includes reward traces, a null-objective control, and a cleaner human-evaluation protocol. Also, the manuscript has unusual text artifacts (author names, file paths) that suggest it was not carefully proofread; this should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Strong engineering paper with a clean human-preference result. The two-stage idea—first use an mLLM to rewrite the prompt against DSG/CMQ rubrics, then Bayesian-search seeds and CFG against a multi-objective reward—is not radically new in its parts, but the integration is new and the Best-of-Random baseline is a good move. The 69% vs 14% win rate at 100 runs and the 42% vs 8% against Best-of-Random are real evidence that the pipeline, as a whole, finds videos people prefer.\n\nWhat's missing is any direct look inside the optimizer. The reward function (RAHF, UVQ, VTA) is never reported during search. We don't see per-iteration best values, convergence curves, or a correlation between the reward and human preference at the candidate level. The paper argues in the appendix that V-Bench++ metrics are flat, which is fine, but then it optimizes a different set of metrics without showing they move. Given the VTA loop is self-referential (the mLLM writes the questions and answers them), the validation at 87% overall agreement helps, but it doesn't prove the search signal is strong enough to guide the optimizer. This is the weakest link.\n\nMinor weaknesses: only two annotators, each on disjoint sets, so no inter-annotator agreement can be measured. No code or data release. The prompt optimization ablation shows 27% vs 5% preference, but 68% ties—so the effect is real but modest.\n\nOn balance, the core claim—that the full pipeline beats random search—is supported by the human study. The missing objective traces are the kind of thing that would be fixed in a rebuttal or a revision, not a reason to reject. It's a practical contribution for anyone trying to reduce trial-and-error in production I2V, and the methodology section is clear enough to reproduce.\n\nI'd send this to reviewers. It's not a breakthrough, but it's a solid, well-scoped system paper with an honest evaluation.","headline":"Solid two-stage optimization for I2V with a real human-preference win, but the search objective is never shown to be moving—worth peer review with a request for objective traces.","tokens_in":14427,"tokens_out":2760,"would_cite":true,"duration_ms":24291,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a two-stage closed-loop optimizer—prompt refinement through an mLLM rubric plus Bayesian search over seed and guidance scale—beats unguided trial-and-error for image-to-video generation in human preference tests.","keywords":["agentic optimization","image-to-video generation","prompt optimization","Bayesian optimization","human preference evaluation","video-text adherence","Davidsonian Scene Graph","video generation controllability"],"falsifier":"Re-run the 100-generation comparison, logging the raw reward values for every candidate, and compare the Bayesian search's top-1 against a best-of-random top-1 selected by the same metric ranking. If the reward values show no separation between conditions or if the best-of-random win rate matches the Bayesian win rate, the central claim that the search itself drives the improvement would be falsified.","tokens_in":13314,"feed_emoji":"🎬","tokens_out":8692,"duration_ms":72372,"temperature":0.7,"pith_summary":"The paper tries to establish that a closed-loop optimization framework can turn black-box image-to-video generation from a trial-and-error lottery into a predictable, goal-directed process. It does so with two coupled stages: an iterative prompt-refinement loop that uses a multimodal large language model (mLLM) to generate a yes/no rubric (Davidsonian Scene Graph questions for semantic content, Common Mistake Questions for artifacts), and a Bayesian search that co-optimizes random seeds and classifier-free guidance scales. A new Video-Text Adherence (VTA) score, derived from the same rubric, is part of the multi-objective reward guiding both stages. If the paper is right, professional users can get consistently better-aligned videos under the same compute budget, which is what the human-preference win rates up to 69% are meant to show.","feed_headline":"Closed-loop video search beats random trial-and-error","feed_subtitle":"A two-stage optimizer tunes prompt, seed, and guidance scale, winning up to 69% of human preference tests.","key_machinery":"The load-bearing mechanism is the generator-critic loop. An mLLM constructs a per-prompt tree of yes/no questions; a video is scored by the fraction answered yes, with each child question zeroed out whenever its parent is answered no. That recursive masking makes foundational elements (the presence of an object, say) carry disproportionate weight, because failing them cascades to all descendants. The resulting VTA score is combined with perceptual-quality and motion-consistency measures into a multi-objective reward, which drives both prompt rewriting and a Bayesian co-optimization of seeds (treated as a categorical variable) and classifier-free guidance scales.","core_discovery":"The paper's central claim is that black-box image-to-video generation can be steered reliably by a closed-loop, two-stage optimizer rather than brute-force sampling. Stage one repeatedly rewrites the prompt, scoring each output against an mLLM-generated tree of yes/no questions—DSG questions for semantic content and CMQ questions for artifacts—and keeps the prompt with the highest alignment. Stage two keeps that prompt fixed and runs Bayesian optimization over random seeds and classifier-free guidance scales, guided by a multi-objective score that includes a new Video-Text Adherence (VTA) metric built from the same question trees. In head-to-head human preference tests, the framework's top output beats a randomly selected baseline output in 60–69% of comparisons at a 100-generation budget, and beats a Best-of-Random baseline (random search plus identical metric-based selection) 42% to 8% under RAHF ranking. The paper reads these results as evidence that the search itself, and not merely post-hoc metric selection, is doing the work.","pith_inferences":["If the VTA hierarchy is doing the work, an ablation that replaces recursive masking with a flat average of question scores should reduce the win rate; the paper does not report this comparison.","The same generator-critic loop could optimize for objectives the current rubric does not cover—narrative coherence, emotional tone, physical plausibility—by swapping in new question categories; the paper points to this direction but does not demonstrate it.","Because the paper only reports human preference for the top-1 selection, it leaves open whether the full ranking produced by the reward suite is reliable; a rank-correlation study between VTA/quality scores and human ratings would clarify how much of the gain is attributable to each stage.","The near-flat automated benchmark scores reported by the paper suggest that standard public metrics may systematically under-report improvements that human raters detect, which would make the framework look weaker on automated leaderboards than it is."],"forward_implications":["At a 100-generation budget, the top video from the agentic pipeline is preferred by human raters over a randomly chosen baseline video in roughly 60–69% of comparisons; at 10 generations the win rate is 42–47%.","Against the stronger Best-of-Random baseline—random search followed by the same artifact filters and metric-based selection—the framework still wins 42% to 8% (RAHF ranking), indicating the value comes from the search trajectory and not just from final selection.","Prompt optimization alone, with hyperparameters held fixed, makes human raters prefer the optimized-prompt video 27% of the time versus 5% for the original prompt, with most pairs tied.","Because the framework is a model-agnostic wrapper with modular evaluators, the same closed-loop recipe transfers to other black-box video generators and can incorporate alternative quality objectives without retraining."],"supporting_citations":[{"why":"Supplies the Davidsonian Scene Graph question-generation method that the prompt optimizer adapts and extends to image-to-video.","marker":"[11]"},{"why":"Introduces Bayesian optimization, the search method used to co-optimize seeds and CFG scales.","marker":"[28]"},{"why":"Provides the mixed continuous/categorical Gaussian-process bandit algorithm used to handle seeds as unordered variables.","marker":"[34]"},{"why":"Provides the rich-human-feedback quality model used as one of the multi-objective reward signals.","marker":"[35]"},{"why":"Provides the universal video quality model used as another reward signal and ranking metric.","marker":"[36]"},{"why":"Supplies the V-Bench++ image-to-video benchmark prompts, data, and automated metric suite used for evaluation.","marker":"[37]"},{"why":"Is the video-understanding mLLM whose VQA agreement with human labels (87% overall) is used to justify it as the automated evaluator.","marker":"[33]"}],"fun_headline_variants":["Agentic optimization beats trial-and-error for I2V generation","Closed-loop AI video tuning wins up to 69% of human tests","Two-stage agentic optimizer steers black-box video models","Bayesian-guided agentic search for reliable video synthesis","Beyond random: agentic framework boosts image-to-video adherence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise the argument stands on is that the automated scores steering the search (the quality models plus the VTA rubric) capture the same quality differences human raters see; if those scores are flat or noise-dominated across seeds and guidance scales, then the Bayesian search has no signal to exploit and the reported human-preference advantage is unexplained.","fun_headline_variants_meta":{"raw":{"variants":["Agentic optimization beats trial-and-error for I2V generation","Closed-loop AI video tuning wins up to 69% of human tests","Two-stage agentic optimizer steers black-box video models","Bayesian-guided agentic search for reliable video synthesis","Beyond random: agentic framework boosts image-to-video adherence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1605,"prompt_tokens":1033,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":649,"tokens_out":572,"duration_ms":5048,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:09:54.171297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 100-generation comparison, logging the raw reward values for every candidate, and compare the Bayesian search's top-1 against a best-of-random top-1 selected by the same metric ranking. If the reward values show no separation between conditions or if the best-of-random win rate matches the Bayesian win rate, the central claim that the search itself drives the improvement would be falsified.","supporting_citations":[{"cited_title":"Davidsonian scene graph: Im- proving reliability in fine-grained evaluation for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the Davidsonian Scene Graph question-generation method that the prompt optimizer adapts and extends to image-to-video."},{"cited_title":"Rich human feedback for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Provides the rich-human-feedback quality model used as one of the multi-objective reward signals."},{"cited_title":"Rich features for perceptual quality assessment of ugc videos","cited_arxiv_id":null,"evidence_quote":"Provides the universal video quality model used as another reward signal and ranking metric."},{"cited_title":"Gemini 2.5 pro (gemini-2.5-pro-preview-06-05): Our most advanced intel- ligent ai model, 2025","cited_arxiv_id":null,"evidence_quote":"Is the video-understanding mLLM whose VQA agreement with human labels (87% overall) is used to justify it as the automated evaluator."}],"review_version":1}