{"id":"4054af17-bbbc-4b0a-8a5f-feb4af5e8b85","arxiv_id":"2608.09666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An agentic framework and a distilled 3B model evaluate text-to-image and text-to-video generators with roughly 10% of the samples of standard benchmarks, at the cost of only partial accuracy.","lead":"The paper presents an LLM-driven evaluation agent that checks visual generative models by sampling only a few dozen images or videos per question, then plans the next probe based on what it sees. It also introduces a 10,000-record training set and a small open-source model that imitates the agent, so evaluations no longer require a proprietary API.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable results' claim rests on aggregate within-one-tier accuracy with no confidence intervals; per-dimension, several semantic dimensions (Object Class, Human Action, Spatial) fall at or below a uniform random baseline, so the 10x claim may not hold for fine-grained evaluation.","rationale":"The reader's conditional verdict is appropriate. The strongest parts of the paper are the new EA-CoT-10K dataset, the release of EA-3B, and the transparent discussion of low-scoring dimensions; these are real artifacts and the paper does not hide the instability. My concern does not overturn the contribution but narrows the headline claim. The 'comparable results' phrase is too broad when several fine-grained semantic dimensions are at or below random-tier accuracy and the aggregate uses a loose one-tier tolerance without confidence intervals. The efficiency comparison is less problematic: although Table 4 compares one requested dimension against an average benchmark dimension, the per-dimension sample reduction from roughly 290 to 25 samples persists even if all dimensions are requested. A concrete replication with more trials and per-dimension baselines would settle whether the aggregate story is robust. If it confirms the aggregate story, the claim should be made precise; if not, the claim should be qualified. Thus the verdict remains conditional rather than accept or reject, because the core framework, dataset, and distillation are valuable and the failure mode is localized to fine-grained semantic dimensions and to the unqualified phrasing of the claim.","tokens_in":20864,"tokens_out":4900,"duration_ms":48585,"concrete_test":"Run a controlled replication of Table 2 and Table 5 with 100 trials per model-dimension pair (not 10), at both the default budget and 30 prompts, and compute per-dimension Exact/Within-1 with 95% bootstrap confidence intervals against the uniform random-tier baseline. If the CI for any fine-grained semantic dimension (Object Class, Human Action, Spatial Relationship) overlaps 52% Within-1 or 20% Exact, the paper should restrict 'comparable results' to aggregate quality/temporal dimensions or add an explicit per-dimension disclaimer to the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Evaluation Agent 'delivers comparable results' is load-bearing, and it is also the least secure. Accuracy is measured as agreement with a benchmark-derived five-tier label, aggregated over dimensions and reported as Exact/Within-1 over ten trials. Several per-dimension entries are at or below the uniform random baseline (20% exact, 52% within-1 for five tiers): in Table 5 Open-EA average Within-1 is 35.7% for Object Class, 42.9% for Human Action, and 47.1% for Spatial Relationship; in the API-based Table 2, Human Action averages 42.5% Within-1. No confidence intervals are given, and with ten trials a 0/10 exact score has a wide binomial interval, so these entries are statistically indistinguishable from chance. The paper's own prompt-budget ablation in Section 4.1.2 shows the estimate is not stable: raising the budget from 3-9 to 30 prompts improves Within-1 by 15 points on average, while Exact improves only 7.5 points, indicating tier-boundary noise rather than convergence. If a user asks for fine-grained semantic capability, the agent can be no better than guessing on those dimensions. The paper's discussion in Section 4.2 concedes the contrast between global/temporal dimensions and class/relational dimensions; that concession undercuts the unqualified 'comparable results' claim in the abstract. This is not an internal inconsistency; it is a correctness risk in generalizing from aggregate scores to per-request conclusions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Evaluation Agent, an LLM-driven multi-round framework that evaluates text-to-image and text-to-video generative models by decomposing a natural-language request into sub-aspects, generating a small number of prompts, sampling generated outputs, invoking benchmark or VQA-based evaluation tools, and iteratively updating its plan from the observed evidence. It claims to reduce evaluation time and sample count to roughly 10% of standard VBench and T2I-CompBench pipelines while delivering comparable tier-level conclusions. The paper also introduces EA-CoT-10K, a history-conditioned instruction-tuning corpus derived from rollout trajectories, and EA-3B, a 3B-parameter local planning backbone fine-tuned from Qwen2.5-3B-Instruct, with experiments on four in-domain and three out-of-domain T2V generator families plus a set of open-ended user queries.","tokens_in":21190,"tokens_out":5344,"duration_ms":49639,"significance":"If the efficiency and tier-accuracy claims hold, this is a valuable contribution to generative-model evaluation: it reframes evaluation as adaptive evidence acquisition rather than a fixed pipeline, releases a training corpus and a local open-source agent, and explicitly tests transfer to unseen generator families. The paper is strongest where it is concrete: the multi-round protocol, data filtering and training details, cost comparison in Table 4, prompt-budget ablation in Figure 3, and backbone-robustness check are described in sufficient detail for reproduction, and the authors are transparent about the distinction between API-based and local end-to-end costs. The central 'comparable results' claim, however, is currently supported only by aggregate point estimates over ten trials without confidence intervals, and several per-dimension entries in Tables 2, 3, and 5 are at or below the uniform-random baseline. The contribution is therefore promising but needs additional statistical support and a more qualified statement of the accuracy claim before it can be accepted.","major_comments":[{"comment":"The 'comparable results' claim is load-bearing but is reported only as Exact/Within-1 point estimates over ten trials, without confidence intervals or significance tests. This matters because several per-dimension entries are at or below the uniform-random baseline for five tiers (20% exact, 52% within-1). For example, with ten trials a 0/10 exact score has a 95% binomial confidence interval of roughly 0-31%, and a 1/10 within-1 score has an interval of roughly 0.3-44.5%. Entries such as Table 5's Object Class 0%/0% for Latte-1, Table 5's Spatial Relationship 0%/0% for ModelScope, and Table 2's Human Action 10%/10% for Latte-1 are therefore statistically indistinguishable from chance. Since these are precisely the fine-grained semantic dimensions that distinguish generators, the aggregate Exact/Within-1 numbers cannot support the unqualified 'comparable results' claim in the abstract without per-dimension uncertainty quantification and a statement of which dimensions are reliably above baseline.","section":"Section 4.1.2, Tables 2, 3, 5"},{"comment":"The prompt-budget ablation shows that the default budget of 3-9 prompts per round is not stable for percentage-based dimensions. Raising the budget to 30 prompts improves Within-1 accuracy by 15.0 percentage points on average across Latte-1 and ModelScope, while Exact accuracy improves by only 7.5 points. This indicates tier-boundary noise rather than convergence, and it directly weakens the claim that roughly 25 samples per user question deliver 'comparable results' for fine-grained semantic evaluation. The paper should either use the larger budget in the headline comparisons, report accuracy separately for each budget, or explicitly restrict the comparability claim to dimensions where stability has been demonstrated.","section":"Section 4.1.2, Figure 3"},{"comment":"The T2I-CompBench comparison excludes the Spatial Relationships dimension, with the paper stating that its 'score distribution and statistical interpretation are not directly comparable under our small-sample setting.' This selective exclusion narrows the benchmark claim, yet the abstract and Section 4.1.2 state 'comparable results' and 'high accuracy' without this qualification. Please state explicitly in the abstract and conclusion that the T2I comparison covers four of the five T2I-CompBench dimensions and that the excluded dimension is one for which comparability is not claimed.","section":"Section 4.1.1"},{"comment":"The in-domain Open-EA results are not an independent validation of the learned policy because EA-CoT-10K training traces are derived from rollouts on the same four generators (Latte-1, ModelScope, VideoCrafter-0.9, VideoCrafter-2) that Table 5 labels as in-domain. The paper acknowledges this overlap in the text, but the in-domain rows should be clearly labeled as train-distribution matches, and claims about Open-EA generalization should rest on the out-of-domain rows only. Moreover, many out-of-domain rows are at chance level (e.g., Show-1 Spatial Relationship 0%/0%, CogVideoX-2B Spatial Relationship 0%/0%, CogVideoX-5B Spatial Relationship 0%/0%), so the 'partial cross-family transfer' conclusion needs the same uncertainty quantification and per-dimension baseline analysis as the API-based experiments.","section":"Section 4.2, Table 5"}],"minor_comments":[{"comment":"The notation in Eq. (1) writes y_j = e_k(v_j, c_j), but the dynamic selection of the tool k across rounds is not formalized; define k as a function of the round index and history H_t to match the rollout description in Section 3.2.3.","section":"Section 3.1"},{"comment":"The data-filtering thresholds are described qualitatively ('extremely low aggregate scores', 'lowest-quality range'); please provide the concrete thresholds or point to the code that implements them so the corpus construction is reproducible.","section":"Section 3.3"},{"comment":"Figure 3 is difficult to read because the light/dark bars and hatched/solid regions carry two separate distinctions; add direct value labels or a legend that states both the budget and the exact-versus-within-one distinction for each bar.","section":"Figure 3"},{"comment":"Table 6 reports median and mean runtimes over saved traces, but the row labels do not state how many successful saved traces each row contains; adding the trace counts would clarify the stability of the runtime estimates.","section":"Section 4.2, Table 6"},{"comment":"The paper notes that the 986-record T2I companion set is not mixed into the current EA-3B training run; please state explicitly whether this companion set is intended for future T2I Open-EA training or is released only as a resource.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, the genuinely new pieces are solid: EA-CoT-10K, the distilled EA-3B, and the cross-family transfer tests. Releasing the data and model is a real contribution. Second, the efficiency claim is believable, but the 'comparable results' wording overstates what the accuracy tables show. Several per-dimension numbers, especially Object Class, Human Action, and Spatial Relationship, sit near the uniform random baseline for a five-tier guess.\n\nThe Evaluation Agent loop is a good idea. The paper checks it across backbones, reports a prompt-budget ablation, and shows partial transfer to unseen generator families. The cost reduction from thousands of samples to about 25 per dimension is useful and plausible. The honesty about dimensions where the agent fails is appreciated, but it undercuts the unqualified abstract claim.\n\nSoft spots: accuracy is measured as within-one-tier agreement, with point estimates over ten trials and no confidence intervals. The prompt-budget experiment shows that increasing the budget from 3-9 to 30 prompts lifts Within-1 by 15 points on average, so the estimates are not stable. A few dimensions are statistically indistinguishable from chance. The T2I comparison drops Spatial Relationships, which is convenient. The framework itself is prior work from ACL 2025; the incremental contribution is the dataset and the local model. The paper is transparent about this, which I credit.\n\nWeighing it all up: the math is straightforward, the data is reproducible, and the claims, once qualified per dimension, are defensible. This deserves a serious referee. A good reviewer will ask for confidence intervals, a per-dimension breakdown of where the agent fails, and a softened abstract. It's a useful paper for anyone building or using automated evaluation pipelines, and a good reading-group debate about what 'comparable results' means.\n\nRecommendation: send it to review, but expect heavy revision.","headline":"Useful evaluation-agent framework with a solid open release, but the 'comparable results' claim needs qualification per dimension before it's trustworthy.","tokens_in":21730,"tokens_out":2657,"would_cite":true,"duration_ms":23093,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes an LLM-driven evaluation agent that turns a natural-language question into a small number of targeted generation-and-tool rounds, and claims it reproduces benchmark-level capability judgments at about a tenth of the…","keywords":["evaluation agent","visual generative models","text-to-video evaluation","text-to-image evaluation","small-sample tier estimation","LLM as judge","instruction tuning","tool-augmented reasoning"],"falsifier":"Run the agent many times on the same generator and dimension with the default budget, and compare the distribution of predicted tiers against the tier from the full benchmark; if, for a semantic dimension such as human action, the predicted tier jumps across tiers across repetitions, the small-sample tier estimate is not stable enough to support the tenfold-cost-reduction claim.","tokens_in":20676,"feed_emoji":"🤖","tokens_out":9955,"duration_ms":82968,"temperature":0.7,"pith_summary":"The paper tries to establish that evaluating visual generative models can be done the way a human reviewer does it—quickly, adaptively, and with a small number of samples—rather than by running fixed benchmarks that require hundreds or thousands of generations. It proposes the Evaluation Agent, an LLM-driven planner that turns a natural-language question into sub-aspects, generates targeted prompts, samples a few dozen images or videos per dimension, invokes benchmark or VQA tools on each batch, and uses the observations to decide what to probe next. The agent maps raw scores into five capability tiers and claims that its tier predictions agree with full benchmark results within one tier most of the time, while cutting evaluation time to about 10% of the standard pipeline. The paper also claims that this protocol can be distilled into a compact open-source 3-billion-parameter planner trained on 10,042 trajectory records, preserving the reasoning, tool-use, and summary behavior and transferring partially to generator families never seen in training. A sympathetic reader would care because the framework promises cheaper, user-tailored, explainable evaluation of the fast-moving field of image and video generation.","feed_headline":"Agent judges image and video models at a tenth of the usual cost","feed_subtitle":"A planning loop samples a few dozen outputs per question and still lands within one tier of full benchmarks","key_machinery":"The load-bearing mechanism is the propose–sample–evaluate–observe loop run by two cooperating LLM agents: a Plan Agent that selects the next sub-aspect, decides when evidence is sufficient, and writes the final summary, and a PromptGen Agent that designs or selects the generation prompts and VQA questions for each round. Each round samples a small batch of visuals (roughly 3–9 prompts, about 23–26 total per dimension), passes them through a modular Evaluation Toolkit of benchmark or VQA tools, and returns structured observations that condition the next decision. Recorded rollouts are then unfolded into history-conditioned step-level instruction records—tagged as thoughts, tool calls, information, and summaries—forming a 10,042-record corpus that supervises a compact 3-billion-parameter local planner, so the same protocol runs without a proprietary API.","core_discovery":"The central claim, stated on the paper's own terms, is that dynamic multi-round evaluation with a few samples per round is enough to reproduce the conclusions of full benchmarks. The agent's loop is propose–sample–evaluate–observe: for each user query it selects a sub-aspect, has a prompt-design agent create or select prompts, generates roughly 3–9 visuals per round (about 23–26 per dimension in total), runs an evaluation tool, and updates its plan. On a 15-dimension text-to-video benchmark, the API-based agent achieves 44.8% exact-tier and 81.3% within-one-tier accuracy with one proprietary LLM backbone, and 41.5% / 77.2% with another, while using 23–25 samples per dimension versus 290 for the benchmark average; on a four-dimension compositional text-to-image benchmark it reaches 53.8% / 96.3% with about 25 samples versus 3000. The prompt-budget experiment shows that raising the budget to 30 prompts improves within-one-tier accuracy by 15 points on average on percentage-based dimensions, implying a direct trade-off between samples and calibration. The paper further claims that a 3-billion-parameter local planner trained on 10,042 step-level records from recorded rollouts follows the same protocol and retains partial cross-family transfer (41.3% / 77.3% in-domain, 31.1% / 64.9% out-of-domain on exact / within-one-tier accuracy).","pith_inferences":["The near-zero exact-tier accuracy on semantic dimensions such as object class, human action, and spatial relationship suggests the bottleneck is concept coverage rather than planning; a testable extension would be to make prompt diversity per semantic dimension a training feature of the local planner.","The tier-based protocol fits naturally into continuous model monitoring: cheap agent evaluations on each release could flag capability regressions long before a full benchmark run is affordable.","If small-sample tiering is trusted, an adaptive sampler that escalates the prompt budget only when the agent reports borderline evidence would retain accuracy while using even fewer samples than the fixed budget.","The paper hints at a recommendation database built from accumulated evaluations; turning evaluation output into a queryable model-selection service would be a direct next step that the current experiments do not test."],"forward_implications":["A single user query about a model dimension can be answered from about 23–26 generated samples in minutes, rather than 290–3000 samples taking hours.","Because prompts and tools are chosen dynamically, the same framework handles open-ended queries (e.g., how precisely a model can place objects) and returns a detailed textual rationale, not just a score.","The protocol is not tied to one LLM: swapping the planning backbone changes aggregate accuracy by only a few points, so the method can run on whichever capable model is available.","A 3-billion-parameter local planner trained from recorded rollouts preserves the evaluation protocol and transfers partially to unseen generator families, lowering the bar for reproducible, low-cost evaluation.","There is a tunable cost-accuracy knob: raising the per-dimension prompt budget from 3–9 to 30 prompts increases within-one-tier agreement by about 15 points on average, so users can trade time for calibration."],"supporting_citations":[{"why":"Supplies the text-to-video benchmark dimensions, official prompt lists, and the reference protocol whose tier conclusions the agent must reproduce.","marker":"[6]"},{"why":"Supplies the compositional text-to-image evaluation dimensions and the suite of evaluation tools used in the closed-domain T2I experiments.","marker":"[5]"},{"why":"The preliminary version of the Evaluation Agent framework that this paper extends with the open-source training pipeline.","marker":"[2]"},{"why":"Provides the chain-of-thought prompting style that requires the agent to reason before selecting an action.","marker":"[58]"},{"why":"Provides the thought-action-observation loop that structures each multi-round evaluation step.","marker":"[60]"},{"why":"Documents the base model that is fine-tuned into the compact local EA-3B planner.","marker":"[1]"},{"why":"The training framework used to fine-tune the local planner on the step-level instruction records.","marker":"[13]"}],"fun_headline_variants":["AI agent evaluates image and video models at 10% of usual cost","Promptable agent judges visual models with a few dozen samples","Evaluation agent matches benchmarks using 10% of the sampling","Local 3B agent evaluates visual models with human-like understanding","Human-like agent speeds up visual model evaluation by 10x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole efficiency claim rests on the assumption that a few rounds of 3 to 9 prompts per dimension produce a stable estimate of a model's true capability tier; if that estimate is noisy, the 'comparable results' conclusion weakens.","fun_headline_variants_meta":{"raw":{"variants":["AI agent evaluates image and video models at 10% of usual cost","Promptable agent judges visual models with a few dozen samples","Evaluation agent matches benchmarks using 10% of the sampling","Local 3B agent evaluates visual models with human-like understanding","Human-like agent speeds up visual model evaluation by 10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1707,"prompt_tokens":1166,"completion_tokens":541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":782,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":782,"tokens_out":541,"duration_ms":5586,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:06:13.627770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the agent many times on the same generator and dimension with the default budget, and compare the distribution of predicted tiers against the tier from the full benchmark; if, for a semantic dimension such as human action, the predicted tier jumps across tiers across repetitions, the small-sample tier estimate is not stable enough to support the tenfold-cost-reduction claim.","supporting_citations":[{"cited_title":"T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the compositional text-to-image evaluation dimensions and the suite of evaluation tools used in the closed-domain T2I experiments."},{"cited_title":"Evaluation Agent: Efficient and promptable evaluation framework for visual generative models,","cited_arxiv_id":null,"evidence_quote":"The preliminary version of the Evaluation Agent framework that this paper extends with the open-source training pipeline."},{"cited_title":"React: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"Provides the thought-action-observation loop that structures each multi-round evaluation step."},{"cited_title":"Llamafactory: Unified efficient fine-tuning of 100+ language models,","cited_arxiv_id":null,"evidence_quote":"The training framework used to fine-tune the local planner on the step-level instruction records."}],"review_version":1}