{"id":"62253417-a3ee-433b-98cd-e7561f9a6089","arxiv_id":"2611.23890","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multi-agent recursive self-improvement framework improves a video evaluator's temporal quality judgments, achieving top scores on dynamics benchmarks and boosting motion in generated videos when used as an RL reward.","lead":"EVA-RSI is a multi-agent system that improves a video evaluator's ability to judge motion quality through recursive self-improvement rounds, using a constructed exam with fixed targets and Nash bargaining to balance multiple quality dimensions. The resulting evaluator scores highest among 21 tested models on dynamics subsets of public benchmarks and, when used as an RL reward, increases generated motion in videos.","discovery_kind":"new_method","skeptic_critique":{"audit":{"quote":"The main limitation of EV A-RSI is its verifier exam. Controlled edits simplify motion failures and differ from the distribution of real generated videos, where defects interact with scene content and intent. Construction-defined targets describe individual edits. Human-labeled real videos broaden exam coverage; complex, interacting failures remain a target for expansion. Future work should build richer exams from generator failures, validate targets with human judgments, and test transfer across generators.","status":"reviewer_error","question":null,"consequence":null,"objection_id":"O1","published_as":"withdrawn"},"model":"deepseek/deepseek-v4-flash","headline":"Withdrawn: Pith's reviewer misread the paper.","rationale":"","tokens_in":34005,"tokens_out":602,"duration_ms":11414,"concrete_test":"","verdict_should_be":null,"load_bearing_attack":"The paper's words: “The main limitation of EV A-RSI is its verifier exam. Controlled edits simplify motion failures and differ from the distribution of real generated videos, where defects interact with scene content and intent. Construction-defined targets describe individual edits. Human-labeled real videos broaden exam coverage; complex, interacting failures remain a target for expansion. Future work should build richer exams from generator failures, validate targets with human judgments, and test transfer across generators.”","agreement_with_reader":null},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents EVA-RSI, a multi-agent framework for recursive self-improvement of video evaluators. It addresses two challenges: no native verifier for video evaluation and multiple competing objectives. The system uses a constructed-target exam with controlled edits, human-labeled real videos, dimension-specific LLM agents, and a Nash bargaining acceptance rule. Starting from Qwen3-VL-8B, it improves motion evaluation across dimensions, with gains transferring to unseen defects and public benchmarks. The resulting evaluator improves generated motion as an RL reward and in a data-selection pipeline, with human preference gains.","tokens_in":34326,"tokens_out":230,"duration_ms":5529,"significance":"If the results hold, this work demonstrates a practical approach to recursive self-improvement for video evaluation, a domain lacking native verifiers. The multi-agent coordination via Nash bargaining and the constructed-target exam are novel contributions. The extensive evaluation with controls, hidden test bank, and public benchmarks strengthens the claims. The paper is significant for advancing video evaluation and generative model training.","major_comments":[],"minor_comments":[],"open_questions":null,"recommendation":"accept"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper asks whether recursive self-improvement (RSI) can improve a video evaluator when there is no native verifier and when multiple competing objectives must be balanced. The answer is a qualified yes, and the evidence is substantial.\n\nThe core idea is to build a fixed proxy exam with construction-defined targets—controlled edits like reversal, frame dropping, and object disappearance—that supply supervision without human annotation. A small set of human-labeled real videos anchors the exam to perceptual judgment. LLM agents propose improvements for each dimension, a leader coordinates them, and a Nash bargaining rule selects updates that balance progress across dimensions. The loop runs for 170 candidates over 88 rounds in two stages, starting from Qwen3-VL-8B.\n\nThe resulting evaluator, EVA-RSI, improves the main mean on a hidden test bank from 0.706 to 0.906, with the largest gain on smoothness (0.461 to 0.888). Gains transfer to unseen defect types and to public benchmarks, where EVA-RSI achieves the highest observed scores among 21 tested models on dynamics subsets. As an RL reward, it increases generated motion and earns human preference at update 50. A data-selection pipeline incorporating it also increases motion. The paper includes extensive controls: a non-recursive search, a Union SFT baseline, and a recipe replay that isolate the contribution of recursion.\n\nThe main soft spot is the load-bearing assumption that the constructed-target exam is a valid proxy for real motion quality. The authors acknowledge this limitation explicitly: controlled edits simplify motion failures and differ from the distribution of real generated videos. The exam's coverage and fidelity to real failures is what the entire RSI loop depends on. The reliance on Claude Opus 5.5 for agent proposals means the exact search is not reproducible with open models. The loss of the stage-1 state file limits full reproducibility. These are real but acknowledged limitations, and the paper is transparent about them.\n\nThis paper is for researchers working on video evaluation, recursive self-improvement, and multi-objective optimization. It deserves a serious referee. The experimental design is thorough, the controls are informative, and the results are clearly presented. The limitations are honestly stated and point to clear directions for future work.\n\nRecommendation: send to peer review. The paper would benefit from addressing the reproducibility concerns (e.g., testing with open-source agents, releasing all available artifacts) and further validating the exam's generalization to complex, interacting failures, but these are directions for revision, not reasons for rejection.","headline":"EVA-RSI shows that recursive self-improvement can work for video evaluation by using a constructed-target exam and multi-agent Nash bargaining, producing a stronger evaluator that transfers to real generation tasks.","tokens_in":34710,"tokens_out":588,"duration_ms":9566},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper reports that EVA-RSI, a multi-agent recursive self-improvement framework, improves a video evaluator's motion judgment across multiple dimensions, with the main mean on a hidden test bank rising from 0.706 to 0.906, and…","keywords":["recursive self-improvement","video evaluation","motion quality","multi-agent system","Nash bargaining","constructed-target exam","reinforcement learning","video generation"],"falsifier":"If a human study showed that the EVA-RSI evaluator's scores on the constructed-target exam do not correlate with human judgments of motion quality on real generated videos (especially for complex, interacting defects), the central claim would be undermined. The paper acknowledges this limitation and reports that human-labeled real videos are included in the exam to mitigate it, but the exam's coverage of real failure modes remains the load-bearing premise.","tokens_in":35651,"feed_emoji":"🎬","tokens_out":1647,"duration_ms":21467,"temperature":0.2,"pith_summary":"The paper is trying to establish that recursive self-improvement (RSI) can work for video evaluation, a domain that lacks a native verifier and has multiple competing objectives. The authors introduce EVA-RSI, a multi-agent framework that uses a constructed-target exam (with controlled edits like reversal and frame dropping) to provide supervision, and a Nash bargaining rule to balance progress across dimensions like physics, smoothness, and dynamics. Starting from Qwen3-VL-8B, the loop improves motion evaluation, with the largest gain on smoothness (from 0.461 to 0.888), and the gains transfer to unseen defect types and public benchmarks. If true, this would show that RSI can extend beyond math and coding to perceptual tasks where ground-truth verifiers are unavailable, and that the resulting evaluator can improve video generation when used as an RL reward or data filter.","feed_headline":"Authors report RSI framework improves video motion judgment","feed_subtitle":"EVA-RSI raises evaluator scores from 0.706 to 0.906 on hidden test, leads 21 models on dynamics benchmarks","key_machinery":"The central mechanism is a multi-agent RSI loop with three key components: (1) a constructed-target exam that pairs clean clips with clips containing controlled temporal defects (reversal, frame dropping, freezing, etc.) and assigns the clean clip as the pairwise target, providing supervision without human annotation; (2) dimension-specific LLM proposer agents that study the evaluator's errors and propose training recipes, coordinated by a leader agent; (3) a Nash bargaining acceptance rule that accepts a candidate only if all main dimensions improve (or hold once saturated) and no dimension falls below its floor, preventing gains on one dimension from causing losses on another. The exam is fixed within each stage, making candidate scores comparable, and humans redesign the exam between stages to address saturation points.","core_discovery":"The paper claims that EVA-RSI, a multi-agent recursive self-improvement framework, can improve a video evaluator's temporal quality judgments without a native verifier and across multiple competing dimensions. The system builds a fixed proxy exam with construction-defined targets (controlled edits like reversal, frame dropping, object disappearance) that supply supervision without human annotation, plus a small set of human-labeled real videos to anchor the exam to human judgment. Each dimension has an LLM proposer agent that studies errors and proposes fixes; a leader coordinates proposals; a Nash bargaining rule selects updates that balance progress across dimensions. Starting from Qwen3-VL-8B, the loop produces eight champions over 88 rounds, raising the main mean on a hidden test bank from 0.706 to 0.906, with the largest gain on smoothness (0.461 to 0.888). The gains carry over to defect types never seen during the run, and on the dynamics subsets of public benchmarks the evaluator achieves the highest observed scores among 21 tested models. As an RL reward for Wan2.1, it increases generated motion and earns human preference; a data-selection pipeline incorporating it also increases motion.","headline_check":{"rule":"report_a_claim_v1","sent_back":false,"unattributed":{}},"pith_inferences":["The paper's reliance on a constructed-target exam with controlled edits as a proxy for real motion quality is the load-bearing assumption. If this exam does not capture the distribution of real generated video defects (where defects interact with scene content and intent), the RSI loop may optimize for the wrong signal. The authors acknowledge this limitation explicitly.","The approach requires significant human effort in designing the exam, reviewing candidate changes, and revising the curriculum between stages. Scaling this to new domains or larger models may require automating more of the curriculum design.","The paper shows that the evaluator improves motion as an RL reward, but also notes that adherence (prompt following) can degrade with longer training. This trade-off between motion and adherence may need to be addressed for practical deployment.","The comparison against proprietary models on public benchmarks shows EVA-RSI leading on dynamics subsets but trailing on overall mean rank score. This suggests that the gains are concentrated on motion-related metrics, and other dimensions (like text alignment) may not benefit as much."],"forward_implications":["If the EVA-RSI approach is correct, then recursive self-improvement can be applied to perceptual evaluation tasks that lack a native verifier, extending RSI beyond math and coding domains.","The resulting evaluator can serve as an effective RL reward for video generators, increasing motion and earning human preference, which could improve the quality of generated videos.","The evaluator can also be used as a data filter in training pipelines, selecting clips that lead to more motion in the generated outputs.","The Nash bargaining rule provides a principled way to coordinate improvement across multiple competing objectives, which could be applied to other multi-dimensional evaluation tasks.","The gains transfer to unseen defect types and public benchmarks, suggesting that the learned motion judgment generalizes beyond the synthetic exam."],"supporting_citations":[{"why":"Qwen3-VL-8B is the starting model for the RSI loop; the paper shows it misses motion defects.","marker":"[2]"},{"why":"VideoReward is the baseline reward model; the paper shows it reduces motion when used as an RL reward.","marker":"[32]"},{"why":"VBench provides the dynamics preference pairs used as a public benchmark for comparison.","marker":"[22]"},{"why":"VideoScore-Bench provides the dynamics correlation metric used as a public benchmark.","marker":"[17]"},{"why":"Nash bargaining provides the theoretical foundation for the acceptance rule that balances competing dimensions.","marker":"[42]"},{"why":"Wan2.1 is the video generator used in the RL and data-filtering experiments.","marker":"[50]"},{"why":"Flex-RSI is the prior work on recursive self-improvement for vision and robotics tasks that the paper builds on.","marker":"[72]"},{"why":"VideoPhy-2 provides human-labeled real generated videos used in the exam's real-video dimension.","marker":"[5]"}],"cache_read_input_tokens":0,"weakest_assumption_plain":"The constructed-target exam with controlled edits (reversal, frame dropping, etc.) provides a valid proxy for real motion quality that generalizes to real generated videos and human perceptual judgments.","cache_creation_input_tokens":0},"created_at":"2026-11-19T09:29:54.067169+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00","input_kind":"pdf_text","input_chars":96782,"input_total_chars":96782,"passes_missing":[],"published_standing":"established_narrower","published_confidence":"moderate","objection_audit":{"version":"pith-objection-audit@1","counts":{"raised":1,"confirmed":0,"questions":0,"withdrawn":1,"moved_to_minor":0,"math_error_not_certified":0,"withdrawn_reviewer_error":1,"typesetting_minor_dropped":0,"withdrawn_already_addressed":0},"open_questions":[],"standing":{"basis":["first_pass_verdict"],"standing":"established_narrower","confidence":"moderate"},"standing_rule":"1. No standing is published (not_assessed) when the review read only the abstract, when the referee pass or the objection check did not run, or when the first pass gave no verdict. 2. The first pass's verdict sets the start: ACCEPT is established, CONDITIONAL is established_narrower, REJECT is not_supported. 3. Each objection that survived the check against the paper's text caps the standing: one that breaks the central claim caps it at not_supported, one that narrows it at established_narrower, one that weakens it at supported_with_gaps. The standing is the lowest of the start and the caps. 4. A first-pass REJECT stands only if a surviving objection breaks the central claim. Without one, the standing is the lowest cap of the surviving objections, or not_assessed when none survived. 5. The confidence is the lowest of the first pass's confidence and the confidence of each objection that set the standing, and is no higher than moderate while a question is open or part of the paper's text was not supplied. 6. If any objection was withdrawn because Pith's reviewer misread the paper, the confidence is no higher than moderate, whatever the first pass said: a review that misread the paper once may have misread it elsewhere.","input_kind":"pdf_text","nothing_to_audit":false,"bar":{"enabled":false}},"run_record":{"models":["deepseek/deepseek-v4-flash"],"passes":{"reader":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.01059,"tokens_in":77414,"tokens_out":36479,"duration_ms":342675,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56","message_sha256":"ac3a383fa06e44586c62a1e785ece90633381103dea614e4f55a4d9f79b5ceb2","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"referee":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003166,"tokens_in":34326,"tokens_out":230,"duration_ms":5529,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"a852d99aea0ddfd900e1483f76e20777aae3afde9c016d5b8d6e5db01064c728","message_sha256":"b5bdddab399525976978b3cbeaf96d8a4d71c6fc11061ea1f8ec22ed3575e6bb","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"skeptic":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003204,"tokens_in":34005,"tokens_out":602,"duration_ms":11414,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"8f88c0f61488b1e764c317998cddf35fffc33a1f20a3b1be54330d7843386fc7","message_sha256":"83bad608f768ed0c4d2f0661a6657713f7b98649d72374e1b4e89b5a9f2eb771","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"circularity":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003668,"tokens_in":68318,"tokens_out":231,"duration_ms":7546,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"e9946e39d67b76cc60476379bc32abdc44665b5f0c438659796547bb1041a01f","message_sha256":"6a8d0bd550a41c14dde0bad6b866d07f48552de85b90789fc0044407be9014e1","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"desk_editor":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003266,"tokens_in":34710,"tokens_out":588,"duration_ms":9566,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"e7211f6a3d2734c1025ce7e7e6d48f78b0944d0a32d5021f81be63b27721420a","message_sha256":"84897700a89b123a104d5c395891633925a643d573b1061e946b771e20cd731c","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"objection_audit":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.005038,"tokens_in":104510,"tokens_out":2872,"duration_ms":35739,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"6c7a6883d5517cf9e4945d83947200a36dcd152b8b56c32baf7500d8b3841d0a","message_sha256":"a3c1ba5a78f989edef6304dec881a6faa57b24bb7a2c1f20f45e91865f6801cb","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"pith_extraction":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003544,"tokens_in":35651,"tokens_out":1647,"duration_ms":21467,"temperature":0.2,"input_sha256":"cd23fa0c810efb42ba65824311b8eb66a5f1e85de3db6ad756416233ee80763b","prompt_sha256":"e4f56bf568e7854d0a1d2efb6ec96f8697d5132737cdb31cb7fc64acbd923974","message_sha256":"1a3d7ed6418cd30e3fd82f16105608878c3d490bdfaa3683aaf8175b0abef17c","cache_read_input_tokens":0,"cache_creation_input_tokens":0}},"version":"pith-run-record/v1","prompt_set_sha256":"d60f54226f003b7308ab6f0f9f2951c09c726f00cc66fc05af3456f212247e93","configured_pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium"},"basis":{"kind":"full_text","source":"recorded","chars_given":96782,"chars_total":96782},"receipt":{"review_sha256":"38579597157974828614a9fc10b6677c636e7643599c6246b9ac3cf8a6e8c152","rule":"pith-review-sha256/v1","hashed_at":"2026-11-19T09:31:31.177610+00:00","fields":["id","arxiv_id","paper_version","created_at","pipeline_version","model_set","run_record","input_kind","input_chars","input_total_chars","passes_missing","verdict","confidence","novelty_score","correctness_risk","formal_verification","parameter_count","discovery_kind","one_line_summary","lead","skeptic_critique","referee_report","author_rebuttal","desk_editor","pith_extraction","objection_audit","published_standing","published_confidence"],"matches_current":true},"falsifier":"If a human study showed that the EVA-RSI evaluator's scores on the constructed-target exam do not correlate with human judgments of motion quality on real generated videos (especially for complex, interacting defects), the central claim would be undermined. The paper acknowledges this limitation and reports that human-labeled real videos are included in the exam to mitigate it, but the exam's coverage of real failure modes remains the load-bearing premise.","supporting_citations":[{"cited_title":"VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evalua- tion in Video Generation","cited_arxiv_id":null,"evidence_quote":"VideoPhy-2 provides human-labeled real generated videos used in the exam's real-video dimension."},{"cited_title":"Qwen3-VL Technical Report","cited_arxiv_id":"2511.21631","evidence_quote":"Qwen3-VL-8B is the starting model for the RSI loop; the paper shows it misses motion defects."},{"cited_title":"Improv- ing Video Generation with Human Feedback","cited_arxiv_id":null,"evidence_quote":"VideoReward is the baseline reward model; the paper shows it reduces motion when used as an RL reward."},{"cited_title":"VBench: Com- prehensive Benchmark Suite for Video Generative Models","cited_arxiv_id":null,"evidence_quote":"VBench provides the dynamics preference pairs used as a public benchmark for comparison."},{"cited_title":"VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feed- back for Video Generation","cited_arxiv_id":null,"evidence_quote":"VideoScore-Bench provides the dynamics correlation metric used as a public benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Nash bargaining provides the theoretical foundation for the acceptance rule that balances competing dimensions."},{"cited_title":"Wan: Open and Advanced Large-Scale Video Generative Models","cited_arxiv_id":"2503.20314","evidence_quote":"Wan2.1 is the video generator used in the RL and data-filtering experiments."},{"cited_title":"Flex-RSI: Benchmarking Recursive Self-Improvement of Frontier Agents","cited_arxiv_id":"2611.08946","evidence_quote":"Flex-RSI is the prior work on recursive self-improvement for vision and robotics tasks that the paper builds on."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":"pdf_text"},"url":"https://pith.science/accuracy"},"review_version":1}