{"id":"e91b18ee-dc57-4784-8f0b-d7b301da143e","arxiv_id":"2607.19947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ETPDesigner automatically generates multi-page electronic theater programs from scripts using a multi-agent LLM pipeline with a global style anchor and interactive character chat.","lead":"This paper introduces a multi-agent AI system that generates complete electronic theater programs—posters, info pages, and character portraits—from raw scripts, plus an interactive mode where users can talk to animated characters. It also releases a benchmark dataset for this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported superiority rests on VLM-as-a-Judge scores that the Critic explicitly optimizes during generation; without evidence that these scores track human judgment, the central claim is not established.","rationale":"The reader's weakest assumption identified exactly this: the VLM-as-a-Judge scores are assumed to be valid proxies for real theatrical design quality, and optimizing against them transfers to human judgment. The paper's evidence does not support this assumption: the human study is small, does not compare against the automated metrics, and the in-loop optimization uses the same rubric as the final evaluation. This is the most load-bearing concern because every quantitative claim of superiority (Table 1) is filtered through this potentially biased metric. The RAG leakage and identical ablation rows are additional symptoms of the same underlying problem: the evaluation pipeline is not independent of the generation pipeline. The concrete test of correlating VLM scores with human expert ratings would directly test the proxy validity. If the correlation is low, the paper's central claim is unsupported; if high, the concern is resolved. The reader's CONDITIONAL verdict already conditions on obtaining such human-validated metrics, so no change to the verdict is needed.","tokens_in":13923,"tokens_out":4829,"duration_ms":47534,"concrete_test":"Compute the Spearman rank correlation between the average VLM-as-a-Judge scores (Table 1) and average human expert preference scores across the same set of ETPs generated by ETPDesigner, AutoGen, Seedream 4.5, and Nano Banana Pro. Recruit at least 20 professional theater designers to rate or rank a random sample of at least 50 ETPs on the three dimensions. If the rank correlation is below 0.7, or if the human ranking of methods differs from the VLM ranking, the automated metrics are not valid proxies and the superiority claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim—superiority in semantic fidelity, aesthetic consistency, and interactivity—is supported primarily by VLM-as-a-Judge metrics in Table 1. However, the evaluation is circular: §3.2 describes a Critic (A_crit) that scores generated posters on the same three dimensions and eight sub-metrics (Theatrical Aesthetics, Communication Efficacy, Thematic Fidelity) that are later used as the final evaluation metrics in §5.1. The full model iteratively optimizes against this exact scoring rubric, whereas baselines are not optimized against it. This can inflate the reported margins without reflecting genuine quality differences. §5.5 confirms the Critic's necessity, stating that removing it drops VLM scores, so the optimization is indeed chasing the evaluation metric. The only non-circular evidence is a small human study (49 participants), but it does not validate the automated sub-metrics or report any correlation between VLM scores and human preferences. Additionally, the RAG module in §3.2 retrieves reference posters from the same ETP-Pro dataset used for evaluation (§4), creating potential information leakage: the generated poster for a test script may be guided by professional posters of the same or similar scripts. The suspiciously identical ablation rows for 'Ours w/o Style Extr' and 'Ours w/o Asset Comp' in Table 1 further suggest that the VLM metrics are insensitive to major pipeline changes, consistent with metric-gaming. If the VLM scores are not valid proxies for human-judged theatrical design quality, the central claim of superiority collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ETPDesigner, a training-free multi-agent framework that synthesizes electronic theater programs (ETPs) from raw dramatic scripts. The system chains a semantic analyst, an art director, a text-to-image generator, a VLM critic, a style extractor, and a layout compositor; the core poster is generated via an iterative RAG-enhanced Tree-of-Thoughts loop in which a VLM judge scores the output on theatrical aesthetics, communication efficacy, and thematic fidelity. A global style anchor is then extracted and used to generate style-consistent backgrounds and character portraits, which are programmatically assembled. The authors also introduce ETP-Pro, a benchmark of 100 curated programs, and report VLM-as-a-judge scores, objective consistency metrics (CLIP-I, DINOv2, OCR text accuracy), a small human study, and ablations to support the claim that ETPDesigner outperforms existing T2I and general multi-agent baselines.","tokens_in":14260,"tokens_out":4780,"duration_ms":46995,"significance":"If the evaluation is valid, the paper would make a useful contribution by demonstrating that a modular, training-free multi-agent pipeline can produce aesthetically coherent and semantically faithful multi-page theatrical publications, and by providing a domain-specific benchmark for this task. The global style-anchor mechanism and the decoupled asset-generation strategy are reasonable design choices. The construction of ETP-Pro is valuable as a new resource, and the interactive demonstration (animation, speech, persona-grounded dialogue) is an ambitious extension. However, the central numerical claims rest on an evaluation design that is circular in important ways; these concerns are load-bearing and must be addressed before the claimed superiority can be accepted.","major_comments":[{"comment":"The evaluation is circular with respect to the Critic agent. During generation, A_crit scores every candidate poster on exactly the same three dimensions and eight sub-metrics (Theatrical Aesthetics, Communication Efficacy, Thematic Fidelity) that are later used as the primary VLM-as-a-judge evaluation metrics, and Eq. (4) selects the final poster as the argmax of this score. The proposed system is therefore explicitly optimized against the evaluation rubric, while all baselines and several ablations are not. The reported margins may reflect overfitting to this rubric rather than genuine quality differences. To establish the claim, the authors should (a) use an evaluation VLM that was never used during generation, (b) report the correlation between the VLM scores and human judgments, and (c) show that baselines are not disadvantaged by the absence of critic-style optimization.","section":"§3.2, Eq. (4); §5.1, Table 1"},{"comment":"The RAG module queries ETP-Pro to retrieve reference posters during generation, and ETP-Pro is also the benchmark on which all results are reported. The paper describes no train/test split and no exclusion of the target script's own professional assets from the retrieval corpus. Because each ETP-Pro entry is 'strictly' paired with a source script, retrieval can return the ground-truth poster for the very script being generated, directly influencing strategy pruning and visual refinement. Even when the exact match is not retrieved, using the evaluation distribution as the retrieval corpus inflates scores. The authors must re-evaluate on a held-out set that is disjoint from the retrieval database, or at minimum demonstrate that retrieved references exclude the target script and report results after such exclusion.","section":"§3.2, §4, §5.1"},{"comment":"The ablation rows are internally inconsistent. In Table 1, the row 'Ours w/o Asset Comp' is identical to 'Ours' on every one of the ten VLM sub-metrics, yet Table 2 reports that removing asset composition drops DINOv2 from 0.684 to 0.528 and CLIP-I from 0.857 to 0.814. A VLM judge that cannot distinguish a DINOv2 drop of this magnitude is insensitive to exactly the structural property the paper claims to improve. Similarly, 'Ours w/o Style Extr' is identical to 'Ours' on all poster sub-metrics and only changes the whitespace score. This requires explanation: either the ablations were not executed as described, or the VLM metrics are too coarse to support the paper's claims. The authors should re-run the ablations and, if the reported numbers are correct, discuss why the VLM is insensitive to large objective changes.","section":"Table 1 vs. Table 2"},{"comment":"The human evaluation is too thin to validate the automated metrics. It reports preference-level comparisons for 49 participants across four dimensions but provides no per-dimension numeric scores, no inter-rater agreement, no significance tests, and no correlation between the VLM-as-a-judge scores and human preferences. Since the paper's main quantitative claims—Theatrical Aesthetics, Communication Efficacy, Thematic Fidelity—are VLM scores, the authors must demonstrate that those scores are valid proxies for human judgment. Please provide correlation coefficients (e.g., Spearman) between the VLM judge and human ratings, with confidence intervals, and report statistical significance for the human-preference results.","section":"§5.4, Fig. 5"},{"comment":"The baseline comparison is confounded by asymmetric optimization. The proposed system includes a VLM critic that iteratively optimizes outputs on the same dimensions later used as evaluation metrics, whereas the T2I baselines are only given an LLM recaption step and the AutoGen baseline is configured with generic agents. Thus Tables 1 and 2 partly compare a metric-optimized system against unoptimized pipelines. At minimum, the authors should run the baselines with the same number of refinement iterations and the same critic feedback, or justify why this is not possible, and report the performance of baselines under their own self-refinement loops.","section":"§5.1, baselines"}],"minor_comments":[{"comment":"The caption lists 'PosterCraft' as a compared method, but §5.1 defines baselines as Nano Banana Pro, Seedream 4.5, and AutoGen. This inconsistency should be corrected or PosterCraft should be included in the experimental setup.","section":"Figure 4 caption"},{"comment":"The text says '100 unique electronic program' — should be 'programs'. Also, the paper does not describe any train/validation/test split for ETP-Pro, which is critical given the RAG-based method.","section":"§4"},{"comment":"The ETP-Pro statistics state 37 musicals, 63 spoken dramas, 73 dramas, 19 comedies, 8 tragedies, and 61 Chinese plus 39 international works. The genre counts (73+19+8=100) are consistent, but the paper should clarify how the same entries can be classified along multiple axes (e.g., musical vs. drama) and whether the retrieval/evaluation uses these labels.","section":"§5.1"},{"comment":"The text says 'We conducted parallel evaluations across all results using GPT-4o and Gemini-1.5-Pro, with detailed outcomes provided in the Appendix.' As a reviewer, I cannot verify this Appendix because it is not in the submitted manuscript. Please include those results in the supplementary material or main text; otherwise, the claim is unverifiable.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and underexplored task, and the multi-agent design is plausible. However, the evaluation pipeline conflates the optimization target with the measurement instrument and uses the benchmark as both the retrieval source and the test set. These are not merely presentation weaknesses; they affect the validity of the reported superiority. The human study is too small to resolve the issue. A major revision should require a properly disjoint evaluation, a demonstration that the VLM judge correlates with human judgment, and ablation rows that are not identical to the full model. If these cannot be established, the manuscript should be rejected despite the interesting system design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi colleague,\n\nIf you're thinking about this paper, the thing to know: it's a well-organized systems paper for a genuinely new task—generating electronic theater programs (posters, info pages, character portraits) from raw scripts—and it ships a new 100-program benchmark. But the reported wins rest on VLM-as-a-Judge scores that the system itself optimizes, so the superiority claim doesn't hold as stated. The paper needs a serious referee but also needs major rework of the evaluation.\n\nWhat's new: the task itself hasn't been addressed before. The multi-agent pipeline (semantic analysis, art direction, RAG-guided ToT generation, style anchor for consistency, programmatic layout) is a reasonable decomposition. The style anchor idea—distilling global style from the core poster and forcing it on downstream assets—is sensible, and the objective CLIP/DINOv2 results show it helps consistency. The ETP-Pro benchmark is a useful resource, even if small.\n\nThe soft spots are in the evaluation. First, the Critic agent in §3.2 scores images on exactly the same three dimensions and eight sub-metrics used as evaluation metrics in §5.1. The full model iteratively selects the poster that maximizes that score; the baselines are not given that optimization. So the large margins in Table 1 might reflect the system chasing the metric, not genuine quality. Second, the RAG module retrieves reference posters from ETP-Pro, the same dataset used for evaluation. There's no mention of a split, so for a test script the system may be pulling the ground-truth professional poster into the Critic's context. That's leakage. Third, the ablation rows for 'w/o Style Extr' and 'w/o Asset Comp' are numerically identical to the full model on nine of the ten VLM sub-metrics, which is suspicious—it either means the VLM judge is blind to those changes or the rows are copy-paste errors. Either way it doesn't inspire confidence in the metric.\n\nThe human study (49 participants) is the only non-circular evidence, but it's small, not statistically detailed, and doesn't validate the automated metrics. There's no code or data release, so I can't verify the numbers.\n\nNet: the system design and benchmark are worth engaging with, but the central claim of superiority is not established. I'd send it to peer review, but with the expectation that the authors will need to show the VLM scores correlate with human judgment, remove the RAG-benchmark overlap, and fix the ablation reporting.\n\nFor your own use: if you work on poster generation or multi-agent T2I, the benchmark and the style-anchor idea might be worth a look. I wouldn't cite the paper's claims yet.\n\nCheers,\n[You]","headline":"Novel domain and benchmark, but the circular VLM-as-a-Judge evaluation means the claimed superiority over baselines is not established.","tokens_in":14796,"tokens_out":4664,"would_cite":false,"duration_ms":41834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a six-agent pipeline, anchored by a style vector extracted from the core poster, can turn a raw dramatic script into a unified, interactive multi-page theater program set without retraining an image generator.","keywords":["Electronic Theater Program","Multi-Agent Collaboration","Stylistic Consistency","Self-Refinement","Multimodal Interaction","Poster Generation","Text-to-Image","Benchmark"],"falsifier":"Present a blind panel of professional theater designers with a set of competing final programs: the one selected by the automated judge, and a lower-scoring candidate from the same run. If experts do not prefer the judged-selected one at a statistically clear rate, the judge-based selection and the critic loop are not the source of the claimed quality.","tokens_in":13796,"feed_emoji":"🎭","tokens_out":5161,"duration_ms":47500,"temperature":0.7,"pith_summary":"The paper tries to establish that generating an Electronic Theater Program — a multi-page set of posters, information pages, and character portraits for a stage production — is a task that a system of specialized AI agents can perform end-to-end from a raw script. Its central move is to generate one 'core poster' first, extract a global style vector from it, and force every subsequent page (readable backgrounds, character portraits) to inherit that style. The authors argue this style-anchor mechanism is what prevents the style drift that plagues direct text-to-image generation, and they support it with a curated benchmark, automated judge scores, and a small human study. They further claim the same assets can be animated, given voices, and connected to persona-grounded dialogue, turning the static publication into an interactive companion. A sympathetic reader would care because it offers a template for automating any multi-page, style-constrained design task, not just theater programs.","feed_headline":"Six AI agents turn a script into a full theater program","feed_subtitle":"A style anchor pulled from the core poster keeps every page consistent and lets users talk to the characters.","key_machinery":"Global style anchor: a two-part visual prior (artistic style descriptor plus dominant color palette) distilled by the Style Extractor agent from the finalized core poster. It is the mechanism that enforces aesthetic uniformity: every subsequent background and character portrait is generated with that prior hard-conditioned into its prompt, so functional pages (which must have negative space for text) and character pages (which must be expressive) remain visually unified with the poster. The paper's ablations show this anchor, not the generation model itself, is responsible for the reported cross-asset consistency.","core_discovery":"On the paper's own terms, the discovery is that 'theatrical abstraction' and cross-page consistency can be engineered without fine-tuning by decomposing the job into six cooperating agents: a semantic analyst that turns the script into themes and character profiles, an art director that formulates visual strategy, a generator, a critic that judges and refines outputs against retrieved professional posters, a style extractor, and a layout compositor. The load-bearing step is the global style anchor: after the critic-guided loop settles on the final core poster, the style extractor distills its artistic style and dominant color palette into an immutable prior that conditions all background and","pith_inferences":["The orchestration pattern should transfer to any multi-page design artifact with one hero visual and several subordinate functional pages—festival brochures, exhibition catalogs, game art books—where a single style anchor would play the same unifying role.","Because the whole program inherits the core poster's style, a cheap practical upgrade is letting a human pick or lightly edit the core poster before the rest of the pipeline runs; the paper does not explore this human-in-the-loop variant.","The talking-character layer suggests an untested accessibility use case: an audio-interactive program could serve visually impaired theatergoers, since the same pipeline already produces voice and persona.","A stress test worth running: feed the system one script twice with different random seeds; if the two resulting programs differ wildly in style, the style anchor is under-constrained by the script and the 'faithfulness' claim needs qualification."],"forward_implications":["From one script, the full program package—poster, readable synopsis background, and character portraits—can be produced without retraining the underlying text-to-image model.","The style anchor ties all pages together: as long as the core poster is right, backgrounds and portraits inherit its look, which means improving the core poster is the highest-leverage improvement to the whole set.","Asset-level decoupling plus deterministic layout avoids garbled typography, so dense professional text such as cast lists and schedules can be rendered without bespoke text-generation.","The interactive layer (animated portraits, synthesized voices, persona-grounded dialogue) is a direct byproduct of the generated assets, so the same pipeline yields both a static program and a conversational character.","Removing any of the four identified components (expert mapping, critic loop, style extractor, asset composition) measurably degrades the reported quality, which the paper treats as evidence that the multi-agent decomposition is doing real work."],"fun_headline_variants":["Six AI agents turn a script into an interactive program you can talk to","No fine-tuning: six agents make consistent interactive theater programs","Script in, interactive program out—six AI agents make it happen","ETPDesigner: multi-agent AI crafts interactive theater programs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole reported superiority rests on the automated critic's 1–10 ratings of theatrical aesthetics, communication efficacy, and thematic fidelity being faithful proxies for what human designers and audiences value—and on the iterative loop that pushes those ratings up actually improving the output rather than just gaming the judge.","fun_headline_variants_meta":{"raw":{"variants":["Six AI agents turn a script into an interactive program you can talk to","No fine-tuning: six agents make consistent interactive theater programs","Script in, interactive program out—six AI agents make it happen","ETPDesigner: multi-agent AI crafts interactive theater programs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001754,"raw_usage":{"total_tokens":6771,"prompt_tokens":759,"completion_tokens":6012,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":5939}},"tokens_in":503,"tokens_out":6012,"duration_ms":40411,"temperature":1.0,"reasoning_tokens":5939,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:11:22.955406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present a blind panel of professional theater designers with a set of competing final programs: the one selected by the automated judge, and a lower-scoring candidate from the same run. If experts do not prefer the judged-selected one at a statistically clear rate, the judge-based selection and the critic loop are not the source of the claimed quality.","supporting_citations":[],"review_version":1}