{"id":"2247d106-08b3-4bc1-b899-b39c6374611b","arxiv_id":"2507.19232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-driven, event-based pipeline plans and animates multiple humans acting coherently in 3D scenes, with a new 40-case benchmark and ablations showing gains over simpler prompting baselines.","lead":"This paper builds a pipeline that uses large language models to plan coordinated everyday activities for several animated humans in a 3D indoor scene, then synthesizes their motions. It also introduces a small benchmark and user study to argue that this event-based approach scales better than simpler LLM prompting strategies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark evaluates single next-step plans for one character, not long-term multi-character storytelling; the headline scalability claim is not yet supported.","rationale":"The reader identified the self-authored benchmark as the weakest assumption. I agree that benchmark validity is central, and I sharpen the concern: even if the benchmark is internally consistent, its design tests single next-step plan selection for one character under permissive pass criteria, not the long-term multi-character storytelling the paper claims. This is a load-bearing gap because the abstract and conclusion make the scalability and long-horizon claims the main contribution. The internal ablation results (full pipeline beating w/o Event, Object List, and Scene Graph across multiple LLMs; PI gains from area-conditioned sampling) are credible evidence for the modular design, and I am not disputing those measurements. However, they do not establish external validity for the headline claim, and the user study is short and permissively scored. The reader's CONDITIONAL verdict remains appropriate: acceptance should require either independent annotation of the benchmark's expected plans, a trivial-baseline check, or an end-to-end long-horizon evaluation with more than 4-5 events. No objection to the authors' integrity or the internal consistency of the reported experiments is intended.","tokens_in":18343,"tokens_out":4176,"duration_ms":43568,"concrete_test":"Run a trivial baseline on the 40 benchmark cases from Section 4.2: for each case, output any action label in AS∩A_e and the centroid (or a sampled point) of the expected area p~_e. Compute its success rate under the Section 4.2 pass conditions and compare with Table 1 'Ours'. If this baseline passes a substantial fraction (e.g., >50%) of the OA/RC/SS cases, the pass criteria are too loose to support the scalability claim. If it fails clearly, the benchmark does discriminate and the reported comparisons stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4 claim long-term motions with more than 4-5 characters in multi-room scenes, robust to LLM choice. The evidence is the 40-case benchmark (Section 4.2) and the user study (Section 4.4). But the benchmark does not run a sequence of events or coordinate multiple characters: it tests only the next plan tau_g=(a_g, p~_g or p_g) for a fixed target character c1 given a mock history H. Pass conditions are permissive: (1) any plan is generated, (2) a_g is in AS∩A_e where A_e is a set of acceptable labels, and (3) the position lies inside a semantically defined area. If A_e is broad, as the notation suggests, a planner can pass by emitting any label in that set and any point in the expected area, without producing a coherent long-term story. The benchmark therefore measures coarse label-and-area retrieval for one character, not the multi-agent, temporally extended coordination that the central claim asserts. The user study also covers only 4-5 events per scenario (Appendix C), which is short-horizon, and participants could select multiple options, weakening the preference signal. Additionally, Table 1 shows strong dependence on LLM size (GPT-4o 0.90 vs Llama-3.1-8B 0.35), which undercuts the 'robust to choice of LLM engines' claim unless 'robust' is defined loosely. The internal ablations are consistent, but they do not reach the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an event-driven framework for generating contextual motions of multiple lifelike humans in a 3D scene. The system decomposes the problem into a sequence of events, uses an LLM-based narrator and event parser to plan high-level behavior, and couples this with area-conditioned position sampling and a low-level motion synthesis module. The authors introduce a 40-case benchmark with criteria for object arrangement, regional context, scene state, and position inference, and evaluate their full pipeline against ablations (w/o Event, Object List, Scene Graph, Direct Inference) across six LLM backbones, adding a user study with 50 participants. The central claims are that the framework generates long-term motions with more than 4-5 characters in multi-room scenes and is robust to the choice of LLM engine.","tokens_in":18613,"tokens_out":2520,"duration_ms":27373,"significance":"If the claims are borne out, the work is a useful system demonstration: it packages LLM planning, scene graph description, area-conditioned location sampling, and motion matching into a single pipeline that produces coordinated multi-character animations in scenes larger than those typically handled by single-character or pairwise motion generation. The paper's strengths are its internally consistent ablation evidence (the full pipeline outperforms all ablations across all tested LLM backbones), the large and consistent gain of the area-conditioned position sampling module on the PI test cases (Table 2), the broad LLM coverage including open-source and commercial models, and the author-provided code, benchmark, and result videos. The internal consistency is a genuine positive. However, the significance of the headline scalability and long-horizon storytelling claims is constrained by the benchmark design, which tests a single next-step plan for a fixed target character rather than a temporally extended multi-character storyline.","major_comments":[{"comment":"The benchmark does not directly support the abstract's claim of 'long-term motions with more than 4-5 characters in various multi-room scale scenes.' Each test case provides a mock planning history H and demands only the next plan for a fixed target character c1; pass/fail requires (1) a nonempty plan, (2) ag in AS ∩ Ae, and (3) position within an expected area. This measures coarse label-and-area retrieval for a single character, not multi-agent coordination over a sequence of events. The scalability discussion tied to Figure 7 varies character counts but does not change the single-plan evaluation. I recommend either adding evaluation over full event sequences with multiple target characters and temporal consistency checks, or visibly narrowing the claims to what the benchmark supports.","section":"Section 4.2, benchmark setup and pass/fail conditions"},{"comment":"The claim of robustness to LLM engines is weakened by the large variation in success rates across backbones: GPT-4o achieves 0.90 total, Qwen2.5-72B 0.71, Llama-3.1-70B 0.72, but Llama-3.1-8B and Qwen2.5-7B drop to 0.35 and 0.39, respectively. If 'robust' means that the pipeline completes without crashing and improves over ablations on each backbone, the claim is defensible; if it means consistent performance, the data contradict it. I ask the authors to state explicitly which notion of robustness they intend and to add a sentence acknowledging the strong model-size dependence.","section":"Table 1 and Section 4.3, robustness to LLM choice"},{"comment":"The user study is a preference vote among ablation outputs, not a head-to-head comparison with prior systems, and it covers only 4-5 events per scenario (and 2 events for MPH11). Allowing participants to select multiple options weakens the preference signal because ties are pooled with genuine preferences; no statistical significance test or inter-rater agreement is reported. This is acceptable as a qualitative check but is not enough to support the 'high scalability' claim, especially since the authors themselves note that the MPH11 result differs from the other scenes. Please report per-scenario vote counts unpooled, add a significance test, and avoid claims beyond the evidence.","section":"Section 4.4 and Appendix C, user study"},{"comment":"All quantitative conclusions rest on a benchmark whose expected plans tau_e = (A_e, p~_e) were authored by the same group that proposes the method. The paper does not report inter-annotator agreement on the expected areas and action sets, nor does it show that the expected plans correlate with a downstream application metric. Because the benchmark is the sole anchor for the framework's superiority claims, I recommend adding (a) a second annotator or agreement measure, (b) an external or at least a more objective grounding test for a sample of cases, or (c) an explicit admission that the reported success rates are relative to the authors' definition and may not generalize to other definitions of plan quality.","section":"Section 4.2, benchmark authorship and expected plans"}],"minor_comments":[{"comment":"The code snippet in Figure 4 contains a typo ('chiars' instead of 'chairs'); please correct it and align the variable names with the narrative text.","section":"Section 3.1.3, code example"},{"comment":"The benchmark uses temperature 0.1 while the user study uses temperature 0.0; the paper should justify this discrepancy or state whether it affects the interpretability of the user study results.","section":"Section A.5.2, evaluation settings"},{"comment":"The sentence introducing Table 4 reads 'In Table 4 Table 4 presents...'; please remove the duplicated phrase.","section":"Section B.2, VLM experiments"},{"comment":"The figure caption does not explain how the colored semantic areas are computed from the object bounding boxes; adding one sentence on the geometric definition of the areas would make the sampling step reproducible.","section":"Figure 12, semantic area representation"},{"comment":"The notation uses both p~_e and p_g with a tilde to denote area-level positions, but the relationship between area-level and coordinate-level outputs is not formally defined; a short formal definition would remove ambiguity.","section":"Section 4.2, benchmark definition"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable system paper, and the ablation comparisons are internally consistent. The main gap is that the benchmark does not actually exercise the long-horizon, multi-character coordination that the abstract promises, and the 'robustness to LLM engines' claim is contradicted by the low success rates of small models. These are fixable with reframing or additional experiments rather than fatal. I would also suggest the editor ask whether the novelty framing ('first to address this problem at scale') is appropriate given existing systems like Digital Life Project and SitcomCrafter; a head-to-head comparison or a clear technical novelty statement would strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful systems paper, but the headline claims outrun the evidence. The stress-test is right: the benchmark measures single-step plans for one character, not long-term multi-character storytelling.\n\nWhat's new and good: the modular design is a real contribution. Splitting planning into narrator and event parser, with a context-centric scene description built from DBSCAN regions and area-conditioned position sampling, is practical and likely to be adopted. The PI ablation shows a large gain, and the whole pipeline beats its ablations across six LLM backbones. That internal consistency is real.\n\nThe soft spots are in the evaluation. The 40-case benchmark is self-authored: expected plans are written by the authors, with no inter-annotator agreement or external baselines. Pass conditions are permissive—any action in the acceptable set and any point in the expected area. Each case asks for the next plan of a fixed character, c1, given a mock history. That does not test the long-term multi-character coordination claimed in the abstract. The user study covers only 4-5 events and permits multiple selections, weakening the preference signal. \"Robust to LLM choice\" is also a stretch when GPT-4o scores 0.90 and Llama-3.1-8B scores 0.35; the relative gains hold, but the absolute spread is large. Code and benchmark are promised but not verifiable from the PDF.\n\nThe paper is not a reject. The method is sound, the writing is clear, and the cited literature is relevant. The appendix's VLM comparison shows they are not hiding weak baselines. I'd send this to review, and ask the authors to temper the scalability claims and either release the benchmark with annotation details or run an external baseline. I'd bring it to a reading group and cite the design, but not the benchmark as evidence for long-horizon multi-agent behavior.","headline":"Useful modular LLM-driven system for multi-character scene animation, but the benchmark evaluates single-character next-step plans and the scalability claims are overstated.","tokens_in":19200,"tokens_out":3961,"would_cite":true,"duration_ms":36051,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An event-driven LLM pipeline can choreograph four or more characters in multi-room 3D scenes, with planning accuracy that holds across several LLM engines.","keywords":["event-driven storytelling","multi-human motion synthesis","LLM-based planning","3D scene understanding","scene-aware behavior planning","area-conditioned position sampling","multi-character animation","benchmark"],"falsifier":"Have independent annotators, who did not design the system, write the expected plan (action label and target area) for each of the 40 benchmark test cases and measure how often they agree; if agreement is low, or if a simple baseline that always picks the most common action and a central area matches the reported success rates, the benchmark is not measuring planning quality.","tokens_in":18109,"feed_emoji":"🎬","tokens_out":5714,"duration_ms":55313,"temperature":0.7,"pith_summary":"This paper is trying to establish that a large language model can direct the long-term, coordinated motion of many virtual humans in a 3D scene if the job is broken into small, sequentially planned events. The framework's narrator writes one event at a time, an event parser turns each event into concrete action labels and target areas, and a motion-matching module animates the characters. The authors claim this event-based decomposition keeps planning accurate as scenes grow to multi-room scale and as the cast grows past four or five characters, and that the results are stable across commercial and open LLMs. They support the claim with a 40-case benchmark covering object arrangement, regional context, scene state, and position inference, plus ablations and a 50-participant user study. If the claim holds, it gives game, VR, and film previsualization a practical way to populate spaces with characters whose behavior emerges from a story rather than from hand-scripted choreography.","feed_headline":"LLM event planner moves 4–5 characters through 3D scenes","feed_subtitle":"Event-by-event planning with area-based positions keeps multi-character stories coherent in multi-room scenes.","key_machinery":"The load-bearing object is the event, an intermediate representation written by a narrator and parsed into $e=(C_e,\\{p_i\\},\\{d_i\\},\\{a_i\\})$: the characters involved, a target 2D position, a target orientation, and an action label for each. Events decompose the holistic spatio-temporal reasoning burden, letting the LLM think about one small interaction at a time while the system as a whole runs events concurrently. The supporting machinery is a scene describer that turns a 3D scene graph plus DBSCAN object clusters into a contextual textual description; an event parser that uses programming-structured prompts with spatial functions such as get distance between and semantic areas; and a low-level motion module that plans paths with windowed cooperative A* and synthesizes motion with motion matching. The key move is that the LLM never has to reason at coordinate level: it selects semantic areas, and exact coordinates are sampled from those areas afterward.","core_discovery":"The central claim is that contextual multi-human motion in 3D scenes stops being one monolithic synthesis problem once you let an LLM plan it as a sequence of events. Each event names a small set of characters, an action, and a rough location; the event parser then grounds that location through spatial-reasoning tools written as Python functions, and area-conditioned position sampling picks exact coordinates. The paper reports success rates of 0.90 total with GPT-4o, 0.74 with GPT-4o mini, and 0.72 with Llama-3.1-70B on its benchmark, with the event pipeline beating an ablation that plans without event decomposition, beating a scene representation that is a raw object list, and beating raw scene-graph input. It also reports that asking the LLM to output coordinates directly fails even on simple localization cases, while the area-conditioned sampling succeeds. The authors present this as the first system to address multi-character contextual motion at this scale.","pith_inferences":["A natural extension the paper leaves implicit is using the narrator as a runtime game director: the same event loop could adapt a storyline to player actions in an interactive 3D environment, not just follow a pre-set instruction.","The benchmark's expected plans were authored by the system designers; an independent check would be to have multiple annotators write expected action labels and areas for the same 40 cases and measure inter-annotator agreement before trusting the pass or fail signal.","The appendix result that vision-based planning performs far worse suggests that current VLMs do not yet replace text scene descriptions for this type of reasoning; a hybrid that feeds visual grounding into the textual description may combine the strengths of both.","If the event decomposition is the real source of the scaling gain, the same pattern could transfer to other multi-agent LLM planning tasks such as embodied robotics or crowd simulation, where monolithic planning currently fails beyond a few agents."],"forward_implications":["A single free-text instruction can drive full-scene storytelling: characters will find objects, use spaces, and interact with each other without per-character scripting.","Because new events are generated only for idle characters, the runtime loop supports long-horizon operation and mid-run user instructions, so interactive rehearsal or gameplay direction becomes possible.","The ablation results imply scene description quality matters more than raw information: an object list loses spatial reasoning, while an exhaustive scene graph hurts performance and token efficiency; a distilled regional description is the better input.","Robustness across LLM engines means the planning module can be swapped for cheaper or local models, with GPT-4o-class models giving the best success rates but smaller open models still producing usable plans.","Area-conditioned position sampling, rather than direct coordinate output, is the component that makes LLM plans executable at the level of precise 3D placement."],"supporting_citations":[{"why":"supplies the automated 3D scene graph construction that the scene describer converts into text.","marker":"[19]"},{"why":"motivates the programming-structured prompt through which the event parser performs spatial reasoning.","marker":"[26]"},{"why":"provides the windowed cooperative A* algorithm used for collision-free multi-character pathfinding.","marker":"[37]"},{"why":"provides the motion matching algorithm that synthesizes the low-level character motions.","marker":"[7]"},{"why":"supplies the HSSD objects used to build the benchmark's House, Office, and Restaurant test scenes.","marker":"[23]"},{"why":"supplies human-scene interaction motions such as sitting and lying for the motion database.","marker":"[14]"},{"why":"is the plot-driven motion generation framework that the paper's scale claim extends beyond.","marker":"[6]"},{"why":"is the autonomous 3D character system that the paper contrasts with for multi-character coordination.","marker":"[4]"}],"fun_headline_variants":["LLM event planner choreographs multi-character 3D scenes","Event-by-event LLM plans moves for up to 5 characters in 3D","LLM breaks 3D group motion into events, not coordinates","Area-aware LLM grounds motion: multi-agent scenes without coordinate overload"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's 40 hand-built test cases, with their expected action labels and target areas, actually measure what scene-aware multi-agent planning should do; if those expected plans do not reflect real user needs, the reported success rates only show internal consistency.","fun_headline_variants_meta":{"raw":{"variants":["LLM event planner choreographs multi-character 3D scenes","Event-by-event LLM plans moves for up to 5 characters in 3D","LLM breaks 3D group motion into events, not coordinates","Area-aware LLM grounds motion: multi-agent scenes without coordinate overload"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1273,"prompt_tokens":954,"completion_tokens":319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":570,"tokens_out":319,"duration_ms":3675,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:57:21.609677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators, who did not design the system, write the expected plan (action label and target area) for each of the 40 benchmark test cases and measure how often they agree; if agreement is low, or if a simple baseline that always picks the most common action and a central area matches the reported success rates, the benchmark is not measuring planning quality.","supporting_citations":[{"cited_title":"Sceneverse: Scaling 3d vision-language learning for grounded scene understanding","cited_arxiv_id":null,"evidence_quote":"supplies the automated 3D scene graph construction that the scene describer converts into text."},{"cited_title":"Code as policies: Language model programs for embodied control","cited_arxiv_id":null,"evidence_quote":"motivates the programming-structured prompt through which the event parser performs spatial reasoning."},{"cited_title":"Cooperative pathfinding","cited_arxiv_id":null,"evidence_quote":"provides the windowed cooperative A* algorithm used for collision-free multi-character pathfinding."},{"cited_title":"Motion matching and the road to next- gen animation","cited_arxiv_id":null,"evidence_quote":"provides the motion matching algorithm that synthesizes the low-level character motions."},{"cited_title":"Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal naviga- tion","cited_arxiv_id":null,"evidence_quote":"supplies the HSSD objects used to build the benchmark's House, Office, and Restaurant test scenes."},{"cited_title":"Stochas- tic scene-aware motion prediction","cited_arxiv_id":null,"evidence_quote":"supplies human-scene interaction motions such as sitting and lying for the motion database."},{"cited_title":"Test scenario employed in the user study for the Restau- rant scene","cited_arxiv_id":null,"evidence_quote":"is the autonomous 3D character system that the paper contrasts with for multi-character coordination."}],"review_version":1}