{"id":"02443bad-0d2f-48e4-b833-3f6a1a277437","arxiv_id":"2507.13285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A three-stage multi-agent system with reflective chain-of-thought, learned layout generation, and iterative visual critique outperforms prior document-to-slide methods on content, coherence, and design metrics, alongside a preference-based evaluator (PREVAL) that reportedly correlates with human…","lead":"The paper introduces RCPS, a multi-agent pipeline that plans, lays out, and iteratively refines slide decks from source documents, plus PREVAL, a learned evaluator that scores presentations on content, coherence, and design. The authors report that RCPS beats existing baselines on both automated and human ratings, though the evaluation rests on a small human study and a metric whose validation may be in-sample.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PREVAL's rho=0.85 is the load-bearing validation for the RCPS superiority claim, but the paper never states whether it is computed on a held-out set disjoint from the 2,000 training pairs; without that, Table 1's PREVAL columns are not independent evidence.","rationale":"The reader identified the same weak spot: PREVAL's validity as an out-of-sample estimator is not established because the paper reports rho = 0.85 without stating a held-out split. I agree that this is the most load-bearing concern. The entire headline result is a performance comparison measured by PREVAL and a small human study; if PREVAL's correlation is in-sample or its training data overlaps with the evaluated systems, the Table 1 advantage becomes training-set evidence rather than independent validation. My read adds one sharper point: the paper also does not disclose which systems generated the 2,000 preference pairs, so circularity is possible in a way the reader's summary only hints at. This is not an accusation of fraud; it is a missing methodological disclosure that can be settled by re-analysis. The secondary issues—small human sample, the FID interpretation, and the Zenodo10K dataset referenced in Appendix C.3 but absent from Section 5.1—are worth fixing but do not replace this primary concern. Because the requested checks are feasible and the flaws are addressable, the verdict should remain CONDITIONAL rather than moving to REJECT or ACCEPT.","tokens_in":20896,"tokens_out":5994,"duration_ms":73379,"concrete_test":"Require the authors to release or precisely describe the PREVAL train/test split and the composition of the 2,000 preference pairs. Concretely: (i) split the 2,000 pairs by source document into train and test sets (e.g., 80/20) and recompute the Section 5.4 Spearman rho on the test pairs only; (ii) state whether any presentation in the training pairs was generated by RCPS or any baseline named in Table 1; if so, retrain PREVAL after removing all RCPS-involved pairs and re-run the Table 1 comparison. A clean pass would show held-out rho remains high (at least ~0.7) and RCPS still leads the retrained metric; a failure would show rho near chance on held-out pairs or RCPS no longer leading, which would invalidate the central superiority claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that RCPS significantly outperforms all baselines (Table 1)—rests on PREVAL as an unbiased proxy for human preference. The only validation offered for that proxy is the reported Spearman rho = 0.85 (Section 5.4, Figure 3), but the manuscript never specifies whether this correlation is computed on a held-out set disjoint from the 2,000 pairwise training comparisons used to train PREVAL (Section 5.1). It also does not disclose which systems produced the two sides of those 2,000 pairs. Two concrete failure modes follow. First, if the 30 human-rated documents overlap with PREVAL's training pairs, or if rho is computed on training data, the correlation is an in-sample fit and says nothing about out-of-sample ranking accuracy; the Table 1 PREVAL columns then do not independently support RCPS superiority. Second, if the preference dataset itself contains RCPS-versus-baseline pairs, then reporting PREVAL's preference for RCPS is at least partially circular: the metric has been trained to reproduce those very preferences. The human evaluation (30 documents, five raters, Appendix G.3) is too small to stand alone, so rho = 0.85 is the load-bearing piece of evidence. The missing held-out disclosure is an omitted support, not a disagreement with consensus; it is directly testable and should be treated as a condition on acceptance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RCPS (Reflective Coherent Presentation Synthesis), a multi-agent framework for automatic document-to-presentation generation, comprising three components: Reflective Chain-of-Thought narrative planning (R-CoT), a Layout Prototype Generator (LPG), and an Iterative Multi-Modal Optimization (IMR) loop. The paper also proposes PREVAL, a preference-based evaluation framework trained on human pairwise comparisons with rationales, and reports that RCPS significantly outperforms four baselines on PREVAL scores and on a small human Likert evaluation, with PREVAL showing a Spearman correlation of 0.85 with human judgments.","tokens_in":21192,"tokens_out":1718,"duration_ms":20485,"significance":"If the reported results hold, the paper makes a useful contribution to automated presentation generation by combining structured narrative planning, learned symbolic layout prototyping, and iterative multimodal refinement; the proposed LDL symbolic layout representation and the editing-primitive-based IMR loop are interesting design choices that could benefit follow-up work. The paper also contributes a preference-based evaluation framework with rationale-enhanced multi-task learning, which is a reasonable direction for reducing reliance on costly human evaluation. However, the empirical support for the central superiority claim is not yet conclusive: the validity of the learned metric is not established with a disclosed held-out evaluation, the human study is very small, and several auxiliary metrics conflict with the claimed advantages. The manuscript would be strengthened by releasing code, data, and a precise evaluation protocol.","major_comments":[{"comment":"The load-bearing validation of PREVAL is the reported Spearman rho = 0.85 between PREVAL Overall scores and human judgments, but the manuscript never states whether this correlation is computed on a held-out set disjoint from the 2,000 pairwise training comparisons described in Section 5.1. If the correlation is computed on the training data or on presentations used to construct the preference dataset, the figure does not establish out-of-sample predictive validity. The authors must disclose the exact split (e.g., number of documents/pairs held out), confirm that the 30 human-rated documents in Appendix G.3 are disjoint from the PREVAL training pairs, and report the correlation on that held-out split; until then, the PREVAL columns of Table 1 do not independently support the RCPS superiority claim.","section":"Section 5.4 / Figure 3"},{"comment":"The central claim that RCPS significantly outperforms all baselines across all dimensions rests on PREVAL scores and a human evaluation of 30 documents rated by five annotators. This evidence base is too thin and too dependent on a single learned metric: the human study has no per-method confidence intervals beyond the reported standard deviations, no analysis of per-document variance, and no error bars for the ablation results in Table 3. In addition, the paper does not report whether the 30 test documents overlap with the domains used for PREVAL training, nor does it give the per-method number of evaluated slides. These details are needed to assess whether the reported superiority is robust rather than an artifact of the small sample.","section":"Table 1 and Section 5.2"},{"comment":"The auxiliary metric FID is inconsistent with the claim of superiority: RCPS has FID = 71.5, which is worse than GPT-4o (68.3) and GPT-4o+VisCoT (64.8), yet the text explains this as 'diverse, content-adaptive visual layouts.' Since FID is typically interpreted as a distributional distance to real presentations, a higher value requires justification; without a perceptual study or a domain-specific calibration showing that higher FID is preferable here, this interpretation is unsupported. The paper should either provide a concrete argument or replace this auxiliary metric with one that is not contradicted by the main claim.","section":"Table 2 and Section 5.2"},{"comment":"The ablation study reports only Overall PREVAL scores with no standard deviations, no confidence intervals, and no statistical test details beyond a star annotation. The text claims that removing R-CoT most significantly impacted Coherence (drop of 0.15) and that replacing LPG with a fixed template degraded Design (drop of 0.20), but the table does not show these per-dimension scores, making the claims unverifiable from the reported data. The authors should report per-dimension results with variances and specify the paired test procedure used for the p < 0.01 claims.","section":"Section 5.3 / Table 3"}],"minor_comments":[{"comment":"The prompt listing in Appendix A contains the phrase 'Diagram flusso di lavoro' and the resulting placeholder text appears to include Italian fragments; please replace with English or a consistent language.","section":"Appendix A.1"},{"comment":"The dataset description mentions 'Zenodo10K subset' in Appendix C.3 but Section 5.1 only describes a 1,000-document RCPS Generation Dataset; these numbers are inconsistent and should be reconciled.","section":"Section 5.1"},{"comment":"The human evaluation section states 'Five actresses evaluated 30 test documents'; this should read 'five annotators' or 'five professional evaluators'.","section":"Section 5.1"},{"comment":"The name 'PREVAL' is inconsistently typeset as 'PREV AL' and 'PREVAL'; please standardize the spelling throughout the manuscript, including the abstract and Figure 3 caption.","section":"Throughout"},{"comment":"The appendix cross-references are inconsistent: Appendix G refers to 'Section H.2' and 'Section H.3' for the evaluation tasks, but the tasks are described in G.2 and G.3; please fix the cross-references.","section":"Appendix G vs. H"},{"comment":"The termination condition 'maximum allowable time limit Tmax' is not defined anywhere in the manuscript; please specify its value or explain how it is set.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The core concern is not the framework design but the evaluative evidence. The PREVAL validation is circular unless a held-out split is disclosed, and the human evaluation is too small to substitute. I would suggest the editor require the authors to state the exact training/validation split for PREVAL, confirm disjointness of the human-rated documents, and release the preference dataset and evaluation code to make the claims auditable. If the held-out correlation is not reported, the paper should be rejected; if it is reported and remains strong, the other concerns are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. Bottom line: this is a serious systems paper with a genuinely assembled pipeline, but the evidence for the central claim is not yet in. The reported Spearman rho = 0.85 between PREVAL and human judgment is the load-bearing validation, and the manuscript never states whether it is computed on a held-out set disjoint from the 2,000 pairwise training comparisons. That omission matters, and the stress-test note lands. What is actually new: the specific combination of R-CoT narrative planning, LPG with LDL symbolic layout generation, the IMR loop, and PREVAL as a rationale-enhanced preference model. LDL is a sensible middle ground between fixed templates and raw coordinate prediction, and the editing primitives make the iterative refinement concrete and testable. The paper is clearly written, the related work is fair, and the limitations section is unusually candid about dependence on foundation models, generalization limits, and resource costs. The soft spots are all in the evaluation. PREVAL's validation is not shown to be out-of-sample, so Table 1's PREVAL columns do not yet independently support the RCPS superiority claim. The human study is small: 30 documents and five raters. No code or data is released. The FID for RCPS is 71.5, worse than GPT-4o+VisCoT's 64.8, yet the text spins this as 'diverse, content-adaptive visual layouts' rather than acknowledging a worse score on that metric. The ablation table has no error bars, so the significance claims rest on unreported variance. Also, the paper does not disclose which systems produced the two sides of the 2,000 preference pairs; if RCPS-versus-baseline pairs are in the training data, using PREVAL to crown RCPS is partly circular. These are addressable problems, not fatal ones. The architecture is plausible, and the limitations section suggests the authors know where the weaknesses are. The paper is for people building document-to-slide systems and learned evaluation metrics for generated presentations. It deserves a real referee, not a desk reject, because the claims are testable and the subfield would benefit from a clear answer. I would send it to peer review with conditions: out-of-sample validation of PREVAL, disclosure of the preference pair sources, release of code and data, corrected FID discussion, and error bars in the ablation. If those come back clean, the paper could be solid.","headline":"A credible pipeline with an underevidenced headline claim: PREVAL's rho=0.85 is the load-bearing validation, and the paper never shows it is out-of-sample.","tokens_in":756,"tokens_out":934,"would_cite":false,"duration_ms":37649,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RCPS, a three-stage multi-agent pipeline, turns documents into presentations that beat all baselines on content, coherence, and design at once, and its PREVAL evaluator tracks human experts at Spearman's rho = 0.85.","keywords":["automatic presentation generation","multi-agent system","reflective chain-of-thought","layout description language","iterative refinement","preference-based evaluation","multimodal quality assessment","document-to-slide synthesis"],"falsifier":"Run the PREVAL-versus-human correlation on a held-out set of pairwise comparisons the model never saw during training, and separately re-run the human 7-point study on a larger, more diverse document sample; if the held-out $\\rho$ falls well below 0.85, or RCPS's 0.7-point human overall margin over GPT-4o+VisCoT disappears under wider sampling, the paper's core comparison loses its support.","tokens_in":20678,"feed_emoji":"📊","tokens_out":5923,"duration_ms":63903,"temperature":0.7,"pith_summary":"This paper tries to establish that high-quality presentation generation can be decomposed into three tractable stages—reflective narrative planning, content-adaptive layout prototyping, and iterative multi-modal refinement—and that a pipeline built this way (RCPS) beats template-based, summarization-based, and direct LLM baselines on all three quality dimensions. It also claims that the preference-based evaluator introduced alongside it, PREVAL, correlates with human judgment at Spearman's $\\rho = 0.85$, making automated assessment a viable substitute for costly human rating. If these claims hold, document-to-slide conversion moves from an ad-hoc text task to a structured planning-and-refinement problem with a trustworthy measurement instrument.","feed_headline":"RCPS beats every baseline on content, coherence, and design","feed_subtitle":"With PREVAL scores tracking human judges at ρ = 0.85, automated evaluation becomes a viable stand-in.","key_machinery":"The load-bearing object is the RCPS pipeline itself, specifically the triplet of R-CoT (a Reflective Chain-of-Thought planner that builds a Thematic Unit Graph and orders narrative modules), LPG (a Transformer encoder-decoder trained by imitation to emit symbolic Layout Description Language sequences instead of pixel coordinates), and the IMR loop (render, critique by VLM-C and LLM-C, edit through parameterized primitives such as move_element, adjust_alignment, and resize_element, repeat). The symbolic LDL representation is what makes the loop work: it gives a compact, learnable starting point that defers pixel-perfect decisions to refinement rather than predicting coordinates in one shot. PREVAL, the paper's evaluation machinery, learns per-dimension scoring functions from pairwise human preferences plus rationales through attention-based multi-task learning.","core_discovery":"On its own terms, the paper's central discovery is that the bottleneck in automated slides is not any single capability, but the absence of a loop in which content, coherent narrative, and visual layout are planned symbolically and then refined against structured visual and logical critiques. RCPS operationalizes this loop: a Reflective Chain-of-Thought planner turns a source document into an ordered outline of slide concepts; a trained Layout Prototype Generator maps each concept to a symbolic Layout Description Language sequence; and an Iterative Multi-Modal Refinement loop renders each slide, has a vision-language critic flag concrete issues like overlap or misalignment and a language critic check coherence, and a refinement agent applies deterministic editing primitives until critique severity falls below a threshold. The paper reports that this system scores 0.73 overall on PREVAL against 0.66 for the strongest baseline, GPT-4o with VisCoT, and 5.4 versus 4.8 in human overall ratings, with the largest margins in Coherence and Design. It additionally reports that PREVAL's scores track human judgments with Spearman's $\\rho = 0.85$, positioning PREVAL as the paper's proposed replacement for purely human evaluation.","pith_inferences":["The paper leaves open whether PREVAL's reported $\\rho = 0.85$ was computed on pairs held out from the 2,000 training comparisons; recomputing the correlation on a disjoint held-out split would test whether the evaluator generalizes or merely memorizes its training distribution.","The paper's own limitation statements imply the approach should be stress-tested on non-English documents and non-Western design conventions, where the current human-annotation base and layout priors may not transfer.","Since the refinement loop's termination threshold and maximum iteration count control cost, a practical deployment question it does not answer is how quality gain scales with compute; Figure 2 suggests there is a knee, so a tuned budget could make the method usable in resource-constrained settings.","PREVAL's rationale-consistency loss points toward a general recipe: evaluation models that are forced to attend to the same features humans cite may generalize better to unseen defects, a hypothesis worth testing in other generation domains."],"forward_implications":["If RCPS is correct, document-to-presentation conversion becomes a planning-plus-refinement problem, and future systems can adopt the same three-stage decomposition.","The reported margins imply one can get near-expert slide quality from an automated pipeline: PREVAL 0.73 versus 0.66 and human 5.4 versus 4.8 over the strongest baseline.","The ablation results imply R-CoT contributes most to Coherence, LPG most to Design, and the iteration loop adds Design refinement beyond one-shot generation.","A preference-based evaluator with rationale alignment can flag presentation defects far better than standard text metrics (F1 0.82 versus 0.45 for ROUGE-L), making quality evaluation cheaper and more diagnostic.","The LPG component is broadly reusable as a standalone layout-prior module for other document-to-visual tasks."],"supporting_citations":[{"why":"DocPres is the strongest pipeline baseline RCPS must beat and the motivating example of inter-stage error propagation.","marker":"Bandyopadhyay et al. (2024)"},{"why":"DOC2PPT establishes explicit layout prediction as the prior approach whose coordinate-level annotation burden LPG avoids.","marker":"Fu et al. (2022)"},{"why":"D2S frames slide generation as query-based summarization, the text-only baseline that RCPS's narrative planning extends.","marker":"Sun et al. (2021)"},{"why":"The PPTC benchmark provides evidence that LLM agents struggle with templates and spatial reasoning, motivating the LPG plus IMR design.","marker":"Guo et al. (2024)"},{"why":"PPTAgent is a current LLM-agent baseline whose template population limits visual flexibility, serving as a key comparison point.","marker":"Zheng et al. (2025)"},{"why":"The fixed-template approach whose rigidity LPG's content-adaptive prototypes are designed to overcome.","marker":"Xu et al. (2025)"},{"why":"Chain-of-thought prompting is the basis that R-CoT extends with reflection for narrative planning.","marker":"Wei et al. (2022)"},{"why":"GPT-4 is the underlying LLM/VLM used in the R-CoT prompts and critics, supplying the base capability the framework refines.","marker":"OpenAI et al. (2023)"}],"fun_headline_variants":["Iterative loop makes RCPS top all slide baselines","RCPS beats GPT-4o slides with reflective planning","Automated slides hit human-level via critiquing loop","PREVAL: automated eval tracks human judges closely","RCPS+loop: 0.73 PREVAL, best in all dimensions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim of RCPS's superiority rests on PREVAL being a faithful stand-in for human quality judgment; that stand-in's validity is supported by a correlation whose training-data overlap is not stated and by human ratings of only thirty documents.","fun_headline_variants_meta":{"raw":{"variants":["Iterative loop makes RCPS top all slide baselines","RCPS beats GPT-4o slides with reflective planning","Automated slides hit human-level via critiquing loop","PREVAL: automated eval tracks human judges closely","RCPS+loop: 0.73 PREVAL, best in all dimensions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2282,"prompt_tokens":927,"completion_tokens":1355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1271}},"tokens_in":543,"tokens_out":1355,"duration_ms":12408,"temperature":1.0,"reasoning_tokens":1271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:26:15.453611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the PREVAL-versus-human correlation on a held-out set of pairwise comparisons the model never saw during training, and separately re-run the human 7-point study on a larger, more diverse document sample; if the held-out $\\rho$ falls well below 0.85, or RCPS's 0.7-point human overall margin over GPT-4o+VisCoT disappears under wider sampling, the paper's core comparison loses its support.","supporting_citations":[],"review_version":1}