{"id":"09a1b1d2-c497-4e7c-b5a7-ca719e719f23","arxiv_id":"2501.03936","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PPTAgent generates presentations by analyzing reference decks and applying LLM-generated edit actions, and PPTEval provides an MLLM-based score for content, design, and coherence.","lead":"This paper introduces PPTAgent, a system that generates slide presentations by editing reference slides instead of writing text into templates, along with a new evaluation framework called PPTEval that scores presentations on content, design, and coherence. If the approach holds up, it could make automatic document-to-presentation tools more visually and structurally useful, not just text-faithful.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PPTEval's coherence rubric explicitly rewards structural slides (opening/ending, speaker/acknowledgments), which is exactly what PPTAgent adds; the headline coherence gain may be an evaluation artifact.","rationale":"The paper is a well-engineered systems contribution with a reproducible pipeline, dataset, and ablations. The method likely does improve generated presentations in ways visible to humans. My concern is not with the approach but with the evidence for the headline quantitative claim. The reader identified the LLM-judge validity issue; I agree and sharpen it: coherence has a concrete scoring-rubric mechanism that rewards the method's signature structural-slide behavior, and human agreement on that dimension is weakest. The proposed test can settle this by removing/adding structural slides or by a human pairwise test. If the gap persists under these perturbations, the claim stands; if not, the paper's conclusion should be reframed as 'improves content/design with better coherence under a structural-completeness rubric,' which is weaker than the stated claim. A conditional acceptance is therefore appropriate: the system is promising, but the evaluation needs to rule out this confound before the superiority claim is accepted.","tokens_in":17071,"tokens_out":3610,"duration_ms":35500,"concrete_test":"Run PPTEval under two perturbations: (1) strip Opening/Ending slides from PPTAgent outputs and re-score; (2) insert generic Opening/Ending slides into DocPres/KCTV outputs and re-score. If the coherence gap collapses in both cases, PPTEval coherence is measuring structural-slide presence, not narrative quality. Additionally, collect human pairwise preferences (e.g., 50 randomly sampled output pairs, 3+ raters blinded to system) between PPTAgent and each baseline; if human ordering does not match the PPTEval ordering with the same margin, the 'significantly outperforms across all dimensions' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that PPTAgent 'significantly outperforms' baselines on Content, Design, and Coherence, measured by PPTEval. The coherence sub-metric shows the largest margin (4.48 vs 3.24-3.57), but its scoring prompt (Fig. 23) awards 4 points for 'basic background information (e.g., speaker or acknowledgments/conclusion)' and 5 for 'detailed and comprehensive background information,' and the extraction prompt (Fig. 20) pulls speaker/date explicitly from opening and closing slides. PPTAgent's Stage I/II is designed to detect and reproduce structural slides (Opening, Ending, Section headers); the baselines have no such mechanism. Thus the coherence score largely measures whether the system emits structural slides, the very thing the method contributes. This is circular for the headline comparison. This concern is compounded by the paper's own agreement data: human-LLM Pearson is only 0.55 for coherence (Table 5), and no human side-by-side between PPTAgent and baselines is reported. Since the claimed 'statistically significant' improvements are not accompanied by significance tests or error bars, the empirical basis for the strongest claim rests on an evaluation whose key dimension is confounded with the method's design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PPTAgent, a two-stage, edit-based system for generating presentations from documents: Stage I analyzes reference presentations by clustering slides into structural and content types and extracting content schemas; Stage II generates an outline and applies executable edit actions to selected reference slides, with a REPL-based self-correction mechanism. The paper also introduces PPTEval, an MLLM-as-a-judge evaluation framework scoring presentations on Content, Design, and Coherence, and releases Zenodo10K, a new dataset of over 10,000 presentations. Experiments compare PPTAgent with DocPres and KCTV across 500 generation tasks, reporting that PPTAgent significantly outperforms both baselines on all three PPTEval dimensions, together with ablations, a score-distribution analysis, self-correction statistics, and a human-LLM correlation study.","tokens_in":17302,"tokens_out":4189,"duration_ms":39110,"significance":"If the empirical claims hold, the paper makes a useful step beyond text-to-slides by formulating presentation generation as reference-guided editing, and it provides valuable public assets in the form of code, a dataset, and an evaluation framework. The ablations, multiple model configurations, and the human correlation study are careful and concrete. However, the headline comparison rests on an author-defined judge whose coherence rubric explicitly rewards structural slides and background information, which are exactly the components PPTAgent adds; the human correlation for coherence is only 0.55, the weakest dimension. The significance of the central claim therefore depends on ruling out an evaluation artifact, which the current evidence does not do.","major_comments":[{"comment":"The headline coherence gains in Table 3 (4.48 vs. 3.24–3.57) are confounded with the method. The PPTEval coherence rubric awards 4 points for \"basic background information (e.g., speaker or acknowledgments/conclusion)\" and 5 points for \"detailed and comprehensive background information,\" and the extraction prompt in Fig. 20 explicitly pulls speaker and date from opening and closing slides. PPTAgent's Stage I/II is designed to detect and reproduce structural slides (Opening, Ending, section headers), whereas DocPres and KCTV have no such mechanism. The coherence score therefore appears to measure, in part, exactly what PPTAgent contributes, making the largest reported improvement an artifact of the evaluation design. I request a human side-by-side comparison between PPTAgent and the baselines, or an alternative coherence evaluation that does not reward the presence of structural slides.","section":"§3, Appendix F (Figs. 20, 23)"},{"comment":"The validity evidence for PPTEval is insufficient for the strength of the central claim. The human-LLM Pearson correlation for coherence is 0.55 (Spearman 0.57), the lowest of the three dimensions, and Appendix B reports Fleiss' kappa of 0.54 for coherence, also the lowest. Since the largest PPTAgent advantage over the baselines is in coherence, the evaluator is least reliable exactly on the dimension that carries the main result. The paper does not report a human evaluation comparing PPTAgent directly with DocPres and KCTV, so the judge's bias on the method-comparison question is not calibrated.","section":"§5.5, Table 5"},{"comment":"The claim of \"statistically significant\" improvements is unsupported by any significance test, confidence interval, or error bar over the 50 presentations per configuration. Moreover, the PPL column shows PPTAgent at 721.54 versus 10–13 for both baselines, a dramatic degradation in the one language-fluency metric reported; the discussion in §4.5 dismisses PPL as inconsistent without a quantitative account. The authors should report paired comparisons (e.g., bootstrap or Wilcoxon over presentations) and explain the PPL discrepancy if the quality claim is to stand.","section":"§4.5, Table 3"}],"minor_comments":[{"comment":"In the DocPres/Qwen2.5LM row, the Success Rate is reported as 100.4%, which is impossible for a percentage; this appears to be a typo for 100.0%.","section":"§4.5, Table 3"},{"comment":"The FID metric is computed using a 64-dimensional output vector, but the feature extractor and the reason for choosing 64 dimensions are not specified; this prevents reproduction of the metric.","section":"§4.4"},{"comment":"The statement that PPTEval's average Pearson correlation of 0.71 \"exceeds the scores of other evaluation methods (Kwan et al., 2024)\" is misleading, since Kwan et al. is MT-Eval, a multi-turn LLM benchmark, not a presentation-quality evaluation method.","section":"§5.5"},{"comment":"The configuration column in Table 8 reads \"Qwen2-VLLM+Qwen2-VLVM\", but the text and the rest of the paper describe the best open-source configuration as Qwen2.5LM+Qwen2-VLVM; the label should be corrected.","section":"Appendix C, Table 8"},{"comment":"The Limitations section acknowledges occasional overlapping elements and imperfect use of visual information, which sits in tension with the blanket claim of superior Design scores; a sentence explaining how the design dimension accommodates these failures would help the reader interpret Table 3.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The coherence circularity is the central risk in this manuscript, and it is compounded by the weak human correlation on that dimension and the absence of significance tests. I would not accept without either a human side-by-side comparison between PPTAgent and the baselines or a version of PPTEval whose coherence criteria do not reward structural-slide presence. The dataset and code release are genuine assets, and the overall direction is publishable if the empirical claims are narrowed or bolstered accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves serious attention but not blind belief in its headline. PPTAgent is a real step forward in framing presentation generation as edit-based, reference-conditioned work, and PPTEval plus Zenodo10K are useful new artifacts. The abstract's 'significantly outperforms' is not yet backed up.\n\nWhat's new: Instead of text-to-template summarization, PPTAgent analyzes reference presentations—clustering slides into functional types, extracting schemas—generates an outline, then applies executable edit actions (replace_span, replace_image, etc.) to HTML-rendered reference slides with self-correction. That is a clear conceptual change from DocPres/KCTV. The ablations are informative: removing HTML rendering cuts success from 95% to 74.6%; removing outline/structure guidance mainly hurts coherence, which makes sense. Zenodo10K (10k+ slides across domains) is a practical resource. PPTEval is a three-axis MLLM judge, and the authors did the right thing by measuring human-LLM correlation (Pearson avg 0.71) and reporting inter-rater kappa (0.59). The limitations section is candid about residual failures, reference dependence, and overlap/design flaws.\n\nSoft spots, in proportion:\n\n- The central comparison uses PPTEval as judge, and the stress-test note lands: the coherence rubric explicitly rewards 'basic background information (e.g., speaker or acknowledgments/conclusion)', which is exactly the structural-slide content PPTAgent is designed to reproduce. Human-LLM correlation on coherence is the weakest (Pearson 0.55). There's no human side-by-side of PPTAgent vs baselines. So the strongest dimension of the claimed win is partly an artifact of the judge's own criteria. This is fixable—acknowledge the confound, report human side-by-side on a subset, or rescore with a rubric that doesn't reward structural slides.\n\n- 'Statistically significant' appears in the results section but no tests, confidence intervals, or error bars are shown. With 50 document/reference pairs per domain, that phrasing is unsupported as written.\n\n- The task setup isn't fully comparable: PPTAgent gets reference presentations; DocPres and KCTV don't. The improvement may come partly from the extra input, not just the editing paradigm. The authors should either feed baselines references or frame PPTAgent as reference-conditioned.\n\nMinor: only 50 documents and 50 reference presentations are used; the code/data repo is promised but I didn't verify it. Citation-wise, the paper engages the right recent work (DocPres, KCTV, Qwen, GPT-4o, judge literature) without padding.\n\nWho benefits: anyone working on document-to-presentation, LLM agents that manipulate structured documents, or LLM-as-judge evaluation. The paper deserves a serious referee; I'd condition acceptance on addressing the evaluation confound and either adding significance tests or softening the significance language.","headline":"A genuinely new edit-based presentation system with a solid dataset and evaluator, but the headline 'significantly outperforms' claim is not yet supported because the main judge shares criteria with the method and significance tests are missing.","tokens_in":17860,"tokens_out":4181,"would_cite":true,"duration_ms":39857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Edit-based agent beats text-to-slides on all three judged quality axes.","keywords":["presentation generation","slide editing","LLM agent","MLLM judge","presentation evaluation","document-to-slides","self-correction","reference-guided generation"],"falsifier":"A direct human preference test in which raters compare unlabeled decks from PPTAgent, DocPres, and KCTV, plus a coherence-only study where opening and ending slides are removed or shuffled, would settle whether the reported gains are real or an artifact of the judge rewarding structural cues.","tokens_in":16873,"feed_emoji":"🎨","tokens_out":4937,"duration_ms":43948,"temperature":0.7,"pith_summary":"PPTAgent argues that presentation generation should be treated as a reference-guided editing task rather than a text-to-slides summarization task. Given an input document and an example deck, it clusters the deck's slides by function, extracts a content schema for each cluster, drafts an outline, and then issues executable commands that rewrite the reference slides one element at a time. The paper also introduces PPTEval, a three-dimension judge (Content, Design, Coherence) using a multimodal LLM, and reports that PPTAgent outperforms rule-based and template-based baselines on all three. This matters because prior methods optimize text in isolation and produce visually flat decks, whereas an edit-based pipeline preserves a designer's layout while replacing content. If accepted, the work reframes the task from generation to curation and offers a reference-free way to measure deck quality.","feed_headline":"Edit-based agent beats text-to-slides on all three quality axes","feed_subtitle":"Reference-guided slide editing preserves layout and lifts content, design, and coherence scores over prior baselines.","key_machinery":"The load-bearing mechanism is the edit-API plus HTML-rendering loop. Instead of generating slides from scratch, the LLM reads a slide rendered as simplified HTML, selects a reference slide by its functional cluster, and writes small executable actions—replace_span, replace_image, clone_paragraph, del_span, del_image—that preserve the original layout and style. A REPL executes the actions, returns errors, and lets the model correct itself until the slide is valid. The other central object is PPTEval, which decomposes quality into slide-level Content and Design and presentation-level Coherence, with detailed scoring prompts; it supplies the numeric evidence for the paper's claims.","core_discovery":"At the paper's center is a workflow that turns slide creation into code. Stage I clusters reference slides into structural and content types and extracts a per-cluster schema of text and image elements; Stage II generates an outline that pairs each new slide with a reference slide and a document section, then emits sequences of API calls such as replace_span and replace_image against an HTML-rendered version of the slide, with a REPL supplying execution feedback for self-correction. PPTEval scores the resulting decks on Content, Design, and Coherence on a 1-to-5 scale, using GPT-4o as judge after converting slides to images. The headline empirical finding is that the Qwen2.5 + Qwen2-VL configuration reaches a 95% success rate and an average score of 3.67, beating DocPres and KCTV on all three judged dimensions, with the largest reported gap in coherence (4.48 versus 3.57 and 3.28).","pith_inferences":["Because human-judge correlation is weakest for coherence (Pearson 0.55), the paper's largest claimed gain is also its least certain; a direct human test focusing on storyline flow, rather than the presence of opening and closing slides, would clarify whether the coherence advantage is substantive.","The cluster-extract-edit cycle is general: the same schema-and-edit loop could be applied to document-to-poster, document-to-infographic, or document-to-storyboard generation, where preserving template design matters.","Since quality is bounded by the reference deck, retrieving the best-matching reference per input document may matter as much as the generation model; a natural extension is retrieval-based reference selection instead of a fixed reference set.","PPTEval's absolute scores on human-authored decks are not reported, so a calibration check is to run the judge on the original Zenodo10K presentations; if human decks score below generated ones, the metric's scale needs recalibration."],"forward_implications":["A deck generated this way inherits the visual layout of a human-designed reference slide, so the model does not need to specify positions, colors, or sizes by hand.","When the reference analysis includes structural slides such as openings, closings, and section headers, coherence scores jump; removing the outline drops coherence from 4.48 to about 3.36, making structural placement the main driver of judged coherence.","Replacing the HTML-rendered representation with the earlier code-interaction format lowers the success rate from 95.0% to 74.6%, so simplifying the verbose XML is what makes reliable editing possible.","The paper's weak correlations of PPL, ROUGE-L, and FID with judged content and design indicate that existing automatic metrics cannot substitute for a purpose-built presentation judge.","Generated quality is bounded by the quality of the reference deck, which means reference selection is itself a quality lever and a potential failure point."],"supporting_citations":[{"why":"Supplies DocPres, the rule-based baseline that PPTAgent must beat on content, design, and coherence.","marker":"Bandyopadhyay et al., 2024"},{"why":"Supplies KCTV, the template-based baseline that PPTAgent must beat and the comparison for judged quality.","marker":"Cachola et al., 2024"},{"why":"Provides the self-correction mechanism that lets the LLM refine edit actions from execution feedback.","marker":"Kamoi et al., 2024"},{"why":"Supplies the alternative code-interaction representation used in the ablation that isolates the value of HTML rendering.","marker":"Guo et al., 2023"},{"why":"Provides the MLLM-as-a-judge paradigm on which PPTEval's scoring is built.","marker":"Chen et al., 2024a"},{"why":"Supplies the presentation design principles behind PPTEval's three dimensions and the human-workflow inspiration for edit-based generation.","marker":"Duarte, 2010"},{"why":"Motivates the use of executable code actions as the interface between the LLM and the slide-editing environment.","marker":"Wang et al., 2024b"},{"why":"Supplies the prior abstractive-summarization approach that PPTAgent contrasts with and whose sampling protocol is reused for the experiment.","marker":"Mondal et al., 2024"}],"fun_headline_variants":["Slide creation as code: edit-based agent tops all three quality axes","Reference-guided editing lifts Content, Design, Coherence","95% success: edit-based agent beats text-to-slides","Turn slide creation into code with reference-guided edits","Edit-based agent beats baselines on all three metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that PPTEval's automated-judge scores are a valid proxy for human quality; the paper's own data show average human correlation of 0.71 but only 0.55 for coherence, and no direct human side-by-side comparison between PPTAgent and the baselines is reported.","fun_headline_variants_meta":{"raw":{"variants":["Slide creation as code: edit-based agent tops all three quality axes","Reference-guided editing lifts Content, Design, Coherence","95% success: edit-based agent beats text-to-slides","Turn slide creation into code with reference-guided edits","Edit-based agent beats baselines on all three metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001913,"raw_usage":{"total_tokens":7460,"prompt_tokens":878,"completion_tokens":6582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":6512}},"tokens_in":494,"tokens_out":6582,"duration_ms":45499,"temperature":1.0,"reasoning_tokens":6512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:43:08.154888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct human preference test in which raters compare unlabeled decks from PPTAgent, DocPres, and KCTV, plus a coherence-only study where opening and ending slides are removed or shuffled, would settle whether the reported gains are real or an artifact of the judge rewarding structural cues.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies KCTV, the template-based baseline that PPTAgent must beat and the comparison for judged quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the presentation design principles behind PPTEval's three dimensions and the human-workflow inspiration for edit-based generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior abstractive-summarization approach that PPTAgent contrasts with and whose sampling protocol is reused for the experiment."}],"review_version":1}