EditDuet: A Multi-Agent System for Video Non-Linear Editing

Bryan Russell; Fabian Caba Heilbron; Gregory Shakhnarovich; Josef Sivic; Marcelo Sandoval-Castaneda

REVIEW 5 major objections 5 minor 1 cited by

EditDuet: A Multi-Agent System for Video Non-Linear Editing

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two LLM agents, one editing and one critiquing, can assemble B-roll timelines from raw footage and a plain-English request.

desk verdict A credible multi-agent video editing system whose headline 'vastly outperforms' outruns the statistics in its own user study. read the letter →

arxiv 2509.10761 v1 pith:FX2BSMJB submitted 2025-09-13 cs.CV

Marcelo Sandoval-Castaneda , Bryan Russell , Josef Sivic , Gregory Shakhnarovich , Fabian Caba Heilbron This is my paper

classification cs.CV

keywords videoeditingnon-linearLLMagentsmulti-agentsystemsB-rollassemblyin-contextlearningLLM-as-a-judgeretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EditDuet argues that video editing—specifically assembling a B-roll sequence to play under a fixed A-roll—can be formulated as a sequential decision-making task and automated by two communicating LLM agents. An Editor agent searches a large footage collection, trims clips, and builds a timeline using standard editing operations; a Critic agent inspects the draft timeline against the original request and issues natural-language revision feedback. The paper introduces a self-supervised exploration stage that synthesizes in-context demonstrations so the two agents learn to communicate without fine-tuning. On real documentary footage, the system reports lower failure rates, higher duration coverage, fewer repeated clips, and stronger human and automatic preference than retrieval-based and single-agent baselines. The paper also proposes an LLM-as-a-judge metric and reports that its automated preferences agree with human raters almost as often as the human raters agree with one another.

What carries the argument

The load-bearing object is the Editor–Critic loop inside a text-grounded non-linear editing environment. The environment exposes four observations to the agents: a transcription of the A-roll, a summary of the whole footage collection, the top-five results of a CLIP-based search_collection tool, and the current timeline. The Editor manipulates the timeline through the editing functions; the Critic compares the timeline to the user request and issues feedback or renders. The second key mechanism is the two-stage exploration process that synthesizes in-context demonstrations—explorer agents, labelers, and scorers—so the Editor learns to follow feedback and the Critic learns to give feasible, a

What would settle it

Run EditDuet on a documentary query whose desired B-roll is defined by abstract or relational content rather than visual categories (e.g., 'shots that convey loneliness,' 'footage that mirrors the rhythm of the voice-over'), and check whether the Critic–Editor loop can recover after search_collection returns unrelated top-5 results; if timelines fail to improve across iterations and coverage, the CLIP retrieval assumption is falsified. A direct diagnostic is to log all search results and count how often the correct clip is never retrieved in the top five at any point during a successful edit.

Watch

Extended reading notes

Core claim

At its center, EditDuet is an iterative loop: the Critic reads the current timeline—a list of clips with durations, descriptions, shot types, and motion labels—and either calls for a render or writes a natural-language corrective note; the Editor then acts through functions such as search_collection, add_to_timeline, remove_from_timeline, and move_clip. Because the Editor never sees the user request directly, the Critic is the sole carrier of intent. Synthetic demonstrations make this division of labor work: explorer agents wander the environment, labelers reconstruct the feedback or user query behind the observed actions, scorers filter for quality, and the survivors become in-context examp

Load-bearing premise

The system's only bridge from raw footage to the timeline is a text-to-clip search that ranks clips by CLIP similarity; if that ranking misses the editing-relevant meaning of a clip, no amount of agent feedback can put the right footage in the cut.

Editorial extensions

If this is right

If the central claim holds, assembling B-roll under a recorded A-roll can be delegated to an automated agent, shrinking the search and rough-cut phase of documentary-style edits.
The iteration between a content-producing agent and a separate intent-holding agent, with synthetic demonstrations generated by exploration, constitutes a template for other tool-using creative tasks beyond video.
LLM-as-a-judge agreement near human–human agreement implies that large-scale, low-cost evaluation of editing quality is feasible, enabling faster iteration on editing systems.
The reported failure-rate reduction from 19.5% to 8.2% suggests that exploration-based in-context examples, not fine-tuning, can substantially stabilize multi-agent tool use.
The system still trails professional human editors (75.1% preference in a supplementary comparison), so the realistic near-term role is assistive rough cuts rather than fully autonomous finishing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because search_collection is the only path from footage to timeline and it ranks by CLIP similarity, the ceiling of the whole system is set by CLIP's ability to capture editing-relevant semantics such as pacing, subtext, and match-cuts; queries for abstract or conceptual B-roll would likely degrade the output even with perfect agent communication.
One testable extension is to swap the retrieval backbone (e.g., a vision-language embedding fine-tuned on editing preferences) and measure whether failure rate and preference improve; the paper's architecture would not need to change.
The judge's keyframe-grid prompt could generalize to evaluating other sequential visual tasks where temporal structure matters, such as slideshow or storyboard generation, but its 80.6% agreement is computed on only this task's distribution.
A natural next step the paper leaves implicit is extending the action space to transitions, speed ramps, and audio, where the Critic's feedback channel already anticipates those operations (e.g., it sometimes suggests transitions the Editor cannot perform).

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper if you work on LLM agents for video. The architecture is genuinely new: an Editor and Critic each with their own tools, plus a two-stage self-supervised exploration that generates synthetic in-context demonstrations for both. That exploration stage is the most original piece—BAGEL-style bootstrapping done multi-agent, with labelers and scorers producing training examples without human annotation. The system is also evaluated on real documentary footage from EditStock, with failure rates, duration coverage, and repetitions reported. That is more than most systems papers do.

The soft spot is the user study behind the central claim. The paper says EditDuet 'vastly outperforms existing approaches' in human preference. The evidence is Table 1: percentages like 13.1% for T2V and 35.1% for the Editor-Critic ablation. These come from 35 people seeing 10 pairs each, but the paper never reports how many pairwise judgments underlie each percentage, nor confidence intervals or significance tests. With 350 total judgments spread over at least five method comparisons and five documentaries, the per-cell counts are probably in the tens. A rate of 13.1% could be 2/15 or 4/30—large binomial uncertainty. The point estimate may be right, but the 'vastly' is not established. The judge-agreement result (80.6% vs 78.7% human-human) is suggestive, but it is the same small sample.

The other limitation is the retrieval ceiling: search_collection returns top-5 CLIP matches, so if CLIP misses the editing-relevant semantics, the agents can't recover. That is a real ceiling, but it applies to all compared methods, so the relative comparison stands. The paper's own limitations section is honest about failure modes like function hallucination and unsupported feedback.

Bottom line: this is a competent systems paper with a plausible architecture and a real evaluation, but the headline claim is statistically shaky. It deserves peer review and likely acceptance after revision—add the pairwise counts, add CIs or significance tests, and soften the language. Also consider releasing code/data; that would materially increase its value. I would cite it if I worked on agentic video editing, and I'd bring it to a reading group on LLM agents.

Referee Report

5 major / 5 minor

Summary. EditDuet proposes a multi-agent LLM system for automatic B-roll assembly in non-linear video editing. The Editor agent (Llama3.1-8B) manipulates an NLE timeline through structured tool calls (search_collection, add/remove/move/switch clips, DONE), while the Critic agent provides natural-language feedback or decides to render. A two-stage exploration process generates synthetic in-context demonstrations to improve agent communication. The system is evaluated on five EditStock documentaries (1458 minutes of footage edited down to 21.5 minutes) against T2V, VisProg, BAGEL, and two ablations, reporting failure rate, time coverage, repetitions, human preference from 35 participants x 10 pairwise judgments, and an automatic GPT-4o judge preference. The paper claims that EditDuet 'vastly outperforms existing approaches' on these metrics, and that the GPT-4o judge agrees with human evaluators 80.6% of the time versus 78.7% human-human agreement.

Significance. If the empirical claims were fully supported, this would be a meaningful step in automating a creative, open-ended task. The strengths are the well-specified NLE environment and tool set, the use of open-source LLMs with structured generation, the synthetic demonstration exploration procedure, the real-world EditStock footage, and the explicit validation of a GPT-4o judge against human preferences using PABAK. The system is largely reproducible from the described components and prompts. However, the central comparative claims are currently supported only by aggregate point estimates with no statistical detail, and the closest prior work (Pardo et al., 2024) is not compared. These issues are fixable but currently limit confidence in the 'vastly outperforms' conclusion.

major comments (5)

[Sec 3.5.1 / Table 1] The headline claim 'vastly outperforms existing approaches' rests on the human-preference column of Table 1. The paper reports only aggregate percentages from 35 participants x 10 pairwise judgments = 350 total, without per-method counts, confidence intervals, or significance tests. For example, the 13.1% human-preference value for T2V could be based on 2/15 or 4/30, giving wide binomial intervals; the same applies to the 35.1% for Editor Critic. Please report the number of pairwise judgments per comparison, exact preference counts, and significance tests (e.g., binomial tests or bootstrap CIs). Also specify how the majority-vote aggregation for the judge-human agreement was computed, including how many participants saw each pair and how ties were handled.
[Table 1 / Sec 4.4] The quantitative metrics (failure rate, time coverage, repetitions) are reported as point estimates with no indication of the number of episodes or runs. The B-roll sequences within the five documentaries are not enumerated, and no standard deviations or per-episode results are given. For instance, failure rates 8.2% (EditDuet) vs. 19.5% (Editor Critic) differ by 11.3 points, but without knowing the denominator we cannot assess whether this is meaningful. Please state the number of user requests per documentary, the number of independent runs per method, and report variance or per-episode breakdowns.
[Sec 3.5.1 / Sec 4.4] The automatic-judge validation and the automatic-preference results in Table 1 may use the same set of timeline pairs. If the 350 human-judged pairs are the same pairs on which the GPT-4o judge's 80.6% agreement was computed, then the automatic-preference column is not an independent confirmation of the method. Please clarify whether the judge was validated and evaluated on disjoint data; if not, split the data or explicitly state the overlap.
[Sec 3.2 / Sec 4.6] The search_collection tool is the only source of footage for the Editor; it retrieves at most five CLIP-similarity results per query. This places an absolute ceiling on the system's content coverage, particularly for editing-relevant semantics such as pacing, mood, or abstract concepts. The paper provides no error analysis of retrieval failures and no diagnostic of how often the Editor's search queries return irrelevant results. While all compared methods share this tool—so the relative comparison is not invalidated—the claim that the system can assemble B-roll timelines from arbitrary footage would be strengthened by a concrete retrieval-failure analysis or at least a discussion of this bottleneck.
[Sec 2.1 / Sec 4.3] Timeline Assembler (Pardo et al., 2024) is described in the related work as taking a video collection and user directive to manipulate an editing timeline, which appears to address the same task. It is not included in Sec 4.3 or Table 1. If it cannot be compared because the implementation or annotations are unavailable, this should be stated explicitly; otherwise, the claim of outperforming 'existing approaches' is incomplete. Please address this closest prior work directly.

minor comments (5)

[Abstract / Table 1] The phrase 'vastly outperforms' overstates the results; Table 1 shows EditDuet ranks top-2 in failure rate and coverage, not top-1 in all metrics. Consider toning down the language or adding the appropriate caveats.
[Table 1] The two percentages in the 'Human Preference' and 'Auto. Preference' columns are not labeled. Define explicitly whether each value is the preference for the row method or for EditDuet. The current text is ambiguous and the reader must infer from the prose.
[Sec 3.4] The term 'learning-based approach' may be misleading, since no parameters are trained; the method selects in-context demonstrations through exploration and scoring. Consider using 'demonstration bootstrapping' or 'ICL-based exploration' to be more precise.
[Sec 4.1] Report the number of B-roll sequences and annotations per documentary. The current description only gives total footage and final-cut duration, making it difficult to assess the number of episodes behind Table 1.
[Supplementary E] The running time of approximately 2.6 minutes on 8 H100s for inference is surprisingly resource-intensive for two 8B-parameter LLMs. A brief explanation of the GPU allocation would help readers understand the practical cost.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EditDuet is an empirical systems report whose central claims rest on external data and human judgments, not on its own definitions or self-citations.

full rationale

This is an empirical systems paper, not a derivation, so the circularity scan finds no step in which an output is equivalent to an input by construction. The central comparative claim is grounded in external evaluation: five EditStock documentaries with manually annotated user requests (Sec. 4.1), and a 35-participant user study producing pairwise human preferences (Sec. 3.5.1); these are reported as raw preference rates in Table 1 and discussed in Sec. 4.4. The GPT-4o 'NLE judge' (Sec. 3.5) is not tuned on the systems it ranks; it is validated by comparing its choices against human majority votes (80.6% agreement vs. 78.7% human-human), and automatic preference numbers are a separate column from human preference. The exploration procedure (Sec. 3.4) uses auxiliary LLM labelers/scorers to generate synthetic in-context demonstrations; that is a self-supervised bootstrapping mechanism for the agents, not a metric defined in terms of EditDuet outputs. The paper's self-citations (e.g., Pardo et al. 2024, Huber et al. 2019) are related-work anchors and are not used to justify a uniqueness theorem or to supply a load-bearing premise. The Limitations section (Sec. 4.6) concedes residual failures (function hallucination, unsupported feedback), which undercuts the strength of 'vastly outperforms' but does not indicate circularity. The human-preference statistics may be underpowered, but that is a robustness concern, not a circularity concern.

Assumptions & free parameters 7 free parameters · 6 assumptions · 3 invented entities

This is an empirical systems paper, so free parameters are operational hyperparameters rather than fitted physical constants. The main unseen costs are the pre-trained models (CLIP, LLaVA, Llama, GPT-4o) and the proprietary EditStock footage. The invented entities are software components of the method; none introduce new physical or mathematical objects.

free parameters (7)

num_icl_demonstrations = 5 per agent
Section 3.4: the exploration stops after collecting five high-quality demonstrations for the Editor and five for the Critic. Chosen by hand.
search_result_limit = 5
Section 3.1: search engine limited to show up to five results at a time.
segment_min_duration = 1 second
Section 3.1: clusters shorter than one second are discarded.
overlap_repetition_threshold = 80%
Section 4.2: a repetition is a pair of sub-clips with at least 80% overlap.
clip_dedup_similarity_threshold = 90%
Section 3.2: shorter overlapping segment is dropped if longer sub-clip has at least 90% CLIP similarity.
demonstration_keep_score = 5
Section 3.4: only pairs scoring 5 are kept as synthetic demonstrations.
caption_frames_per_cluster = 16
Section 3.1: LLaVA-NeXt captions use exactly 16 frames per cluster.

assumptions (6)

domain assumption CLIP similarity is a valid semantic signal for retrieving relevant B-roll footage from a video collection.
Section 3.2, search_collection uses CLIP top-5 retrieval as the only path from footage to timeline; the system's ceiling is CLIP's alignment with editing vocabulary.
domain assumption The caption (LLaVA-NeXt), shot type (MobileNet V3), and camera motion (LSTM on SSD features) classifiers provide accurate text descriptions of every video segment.
Section 3.1; agents reason over these descriptions, so systematic caption errors would propagate into editing decisions.
domain assumption Llama 3.1 8B Instruct with structured generation reliably executes function calls and follows the tool API.
Section 3.2-3.3; both agents use this backbone; failure rates (8.2% final) show residual tool-use errors.
domain assumption GPT-4o judge preferences approximate human preferences for video editing quality.
Section 3.5; validated at 80.6% agreement vs 78.7% human-human, PABAK 0.61 vs 0.57, on a small 35-person study.
domain assumption The authors' manually annotated user requests are representative of how a user would direct a documentary B-roll sequence.
Section 4.1; prompts describe style, content, and duration, but were written by the authors from the ground-truth sequences, which may not match real user language.
domain assumption Transcript of the A-roll audio accurately represents the narration.
Section 3.1; no transcription model or error analysis is given.

invented entities (3)

Editor Explorer, Editor Labeler, Editor Scorer, Self-Reflecting Editor
purpose: Generate and curate synthetic demonstrations for the Editor agent's in-context learning.
Section 3.4; internal auxiliary LLM agents with no external validation; all use the same Llama backbone as the agents they train.
Critic Explorer, Critic Labeler, Critic Scorer
purpose: Generate and score synthetic demonstrations for the Critic agent.
Section 3.4; same internal loop, no independent evidence that the demos generalize beyond the five EditStock projects.
GPT-4o NLE judge independent evidence
purpose: Automatic pairwise preference evaluation of B-roll timelines.
Section 3.5; validated against human preferences (80.6% agreement vs 78.7% human-human, PABAK 0.61 vs 0.57), giving a falsifiable handle outside the paper, though the validation sample is small.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EditDuet: A Multi-Agent System for Video Non-Linear Editing." pith.science (2026). https://pith.science/paper/FX2BSMJB

@misc{pith2026250910761,
  author       = {Pith},
  title        = {Pith review of: EditDuet: A Multi-Agent System for Video Non-Linear Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FX2BSMJB}},
  note         = {Machine review of arXiv:2509.10761}
}

read the original abstract

Automated tools for video editing and assembly have applications ranging from filmmaking and advertisement to content creation for social media. Previous video editing work has mainly focused on either retrieval or user interfaces, leaving actual editing to the user. In contrast, we propose to automate the core task of video editing, formulating it as sequential decision making process. Ours is a multi-agent approach. We design an Editor agent and a Critic agent. The Editor takes as input a collection of video clips together with natural language instructions and uses tools commonly found in video editing software to produce an edited sequence. On the other hand, the Critic gives natural language feedback to the editor based on the produced sequence or renders it if it is satisfactory. We introduce a learning-based approach for enabling effective communication across specialized agents to address the language-driven video editing task. Finally, we explore an LLM-as-a-judge metric for evaluating the quality of video editing system and compare it with general human preference. We evaluate our system's output video sequences qualitatively and quantitatively through a user study and find that our system vastly outperforms existing approaches in terms of coverage, time constraint satisfaction, and human preference.

Figures

Figures reproduced from arXiv: 2509.10761 by the authors.

**Figure 1.** EditDuet’s summary figure. Our system takes as input a user request, an A-roll sequence that is typically interview footage or voiceover recordings, and a real-world video collection (e.g. “The Ovens of Cappoquin” from EditStock). We introduce two large language model (LLM) agents, an Editor and a Critic. These agents interact between themselves through feedback and the non-linear editing (NLE) timeline until the us… view at source ↗

**Figure 2.** Overview of our multi-agent non-linear editing framework. The NLE environment Ω (Sec. 3.1) exposes four observations to the agents: an A-roll transcription 𝑜A, a summary of the video collection 𝑜V , visual search results 𝑜𝑠𝑒𝑎𝑟𝑐ℎ, and the NLE timeline 𝜏. The Editor agent E (Sec. 3.2) modifies 𝜏 based on feedback 𝑓𝑗 from the Critic agent C. The Critic verifies if the timeline satisfies user request 𝑢, then either sugg… view at source ↗

**Figure 3.** Sample run from EditDuet for the documentary “Built By Life”. At each iteration, the Critic observes the current state of the timeline, and produces feedback or decides to render it. The Editor may execute any number of actions until it signals to the Critic that it is finished and ready for the next iteration. All thumbnails are copyrighted and belong to EditStock. SIGGRAPH ’25, August 10–14, 2025, Vancouver, BC, C… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Example outputs for Transcript2Video, VisProg, BAGEL, and EditDuet for the documentaries “Built By Life” and “Ovens of Cappoquin”. Note that our method, EditDuet, is the closest to the desired duration in both samples, as well as the one whose output is semantically cl…

**Figure 7.** Figure 7: Unsupported feedback. The Critic makes a suggestion that is not possible to fulfill given the current set of tools the Editor has access to, namely, adding transitions [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 5.** Figure 5: Function hallucination. The Critic suggests combining sub-clips from a few video files (probably meaning that these should be placed in the timeline consecutively), to which the Editor hallucinates a function called combine_clips. This function does not exist, and thus…

**Figure 8.** Figure 8: Out-of-bounds video sub-clips. The Editor performs a search in the video collection, and then attempts to add one video sub-clip to the timeline. However, the chosen start time and end time for the sub-clip are both higher than the total duration of the source file, so…

**Figure 6.** Figure 6: File hallucination. The Editor attempts to add a file to the timeline regardless of what the Critic has suggested. This file does not exist, and thus, the run fails. SIGGRAPH ’25, August 10–14, 2025, Vancouver, BC, Canada [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
cs.CL 2026-08 conditional novelty 6.0 of 10

GRPB ranks same-task video edits, converts the ranking into zero-sum advantages, and spreads them over editing segments through a lagged, capped allocator, producing a stronger editing agent.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[2023]

Built By Life

Eventfulness for interactive video alignment.ACM TOG42, 4 (2023), 1–10. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. ViperGPT: Visual Inference via Python Execution for Reasoning. InICCV. Bekzat Tilekbay, Saelyne Yang, Michal Adam Lewkowicz, Alex Suryapranata, and Juho Kim. 2024. ExpressEdit: Video Editing with Natural Language and Sketching. InPro...

arXiv 2023
[2024]

BAGEL: Bootstrapping Agents by Guiding Exploration with Language. In ICML. OpenAI. 2023. GPT-4V(ision) Technical Work and Authors. (2023). https://cdn.openai. com/contributions/gpt-4v.pdf Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem, Josef Sivic, Bryan Russell, and Fabian Caba Heilbron. 2024. Generative Timelines for Instructed Visual Assembly. arXiv p...

arXiv 2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [2023]

Built By Life

Eventfulness for interactive video alignment.ACM TOG42, 4 (2023), 1–10. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. ViperGPT: Visual Inference via Python Execution for Reasoning. InICCV. Bekzat Tilekbay, Saelyne Yang, Michal Adam Lewkowicz, Alex Suryapranata, and Juho Kim. 2024. ExpressEdit: Video Editing with Natural Language and Sketching. InPro...

arXiv 2023

[2] [2024]

BAGEL: Bootstrapping Agents by Guiding Exploration with Language. In ICML. OpenAI. 2023. GPT-4V(ision) Technical Work and Authors. (2023). https://cdn.openai. com/contributions/gpt-4v.pdf Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem, Josef Sivic, Bryan Russell, and Fabian Caba Heilbron. 2024. Generative Timelines for Instructed Visual Assembly. arXiv p...

arXiv 2023