{"id":"23d05670-3fa8-4a33-ac4f-a5603bb093a1","arxiv_id":"2505.18240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"REFLEX fine-tunes Phi-3-Mini on synthetic negative presentations to produce reference-free scores and actionable feedback for slide quality across coverage, redundancy, text-image alignment, and flow.","lead":"Researchers built REFLEX, a system that scores presentation slides on four quality metrics and generates written feedback, by fine-tuning a small language model on deliberately corrupted presentations. It introduces a new 8,111-slide dataset and reports that its scores and explanations match human judgments better than common LLM evaluators, though the automatic benchmark is partly circular.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The automated benchmark is circular: REFLEX is trained and evaluated on the same synthetic perturbations, so the central claim rests on a 50-sample human study that likely used the same perturbed slides; a realistic transfer test is required.","rationale":"The central claim requires that the synthetic negative samples used for training are a valid proxy for real presentation quality and that a model trained to recognize these perturbations generalizes to naturally occurring flaws. The reader's weakest_assumption correctly identified this as the load-bearing assumption. I agree with that identification and with the conditional verdict, but I want to sharpen the concern: Table 1 provides no independent evidence because the evaluation set is produced by the same perturbation functions and score mapping used in training; the only non-circular evidence is the human study, which is small and, per Appendix D, appears to include the same perturbed samples, making it vulnerable to the same confound. The paper deserves credit for conducting a human evaluation at all and for showing plausible gains on explanation actionability, but the evidence does not yet establish that REFLEX works on genuinely arbitrary, unperturbed presentations. The proposed test on naturally flawed real decks would settle whether the perturbation-based training captures general quality or merely artifacts of the synthetic pipeline. Since the paper already gates its claim as conditional in the reader's assessment and since the human evaluation offers some support, I do not recommend a harsher verdict, only that the conditional status remain until a realistic transfer test is run.","tokens_in":15363,"tokens_out":3430,"duration_ms":31213,"concrete_test":"Construct a held-out set of 100 real SlideShare or other presentation decks that were not filtered or perturbed by the authors and that contain naturally occurring quality variation. Have at least 3 independent expert raters score each deck on the four metrics using the paper's Likert rubric and rate the actionability of each model's explanations. Then compute Spearman/Kendall correlations with human scores for REFLEX, G-Eval, and Phi3-Eval. If REFLEX does not significantly outperform the best baseline on this realistic set, or if its advantage reverses, the synthetic perturbation labels are not a valid proxy for presentation quality and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that REFLEX outperforms baselines in scores and explanations rests on two legs: Table 1 (correlation with pseudo-ground truth) and Table 3 (correlation with human ratings). The first leg is circular: the pseudo-labels are generated by the same perturbation functions (Section 5.2) and the same score mapping s = 5 - d used to construct the training set, so high correlation only shows the model learned to reproduce those synthetic labels; any baseline that recognized the perturbation type would score well. The second leg is the only independent evidence, but it is thin: 50 samples, 3 annotators, and per Appendix D the annotators were shown 'both original and perturbed slides,' so it may reward models that detect the perturbation degree rather than genuine quality gaps. More critically, the perturbation model assumes each operation degrades exactly one metric and maps degree d linearly to quality; in real presentations, flaws are not isolated (e.g., missing topics also disrupt flow; duplicated slides also affect redundancy and text-image consistency), so a model optimized on these synthetic artifacts may not transfer to naturally occurring faults. If REFLEX's advantage vanishes on unperturbed real-world presentations, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces REFLEX, a reference-free framework for scoring and explaining presentation quality along four metrics: coverage, redundancy, text-image alignment, and flow. The authors curate RefSlides, a dataset of 8,111 SlideShare presentations, generate negative training samples by applying metric-specific perturbations at four degrees with pseudo-ground-truth scores defined as s = 5 - d, and fine-tune Phi-3-Mini with LoRA to produce both scores and actionable explanations. They compare REFLEX with heuristic baselines, G-Eval, and Phi3-Eval, reporting higher correlations with pseudo ground truth (Table 1), higher ROUGE F1 for explanations (Table 2), and generally higher correlations with human ratings on 50 SciDuet presentations (Table 3).","tokens_in":15625,"tokens_out":6301,"duration_ms":67374,"significance":"If the evaluation evidence is validated, REFLEX would be a useful contribution to automatic multimodal presentation evaluation, particularly because it combines score prediction with actionable feedback and does not require a reference presentation at inference time. The paper is also commendable for formalizing four presentation-quality metrics, for providing a modular LoRA-based architecture that can be extended to new metrics, and for including a human evaluation component. However, the central evidence is largely generated by the same perturbation schedule used to construct the training labels, so the real-world value of the method currently rests on a small and partly synthetic human study.","major_comments":[{"comment":"The automatic evaluation in Table 1 is circular as a validation of quality estimation. Section 5.2 defines the training target as sid = 5 - d and generates negative samples from degree-d perturbations, and Table 1 then reports correlations with exactly these pseudo ground-truth labels. Because REFLEX is trained to minimize MSE against these labels, high Spearman and Kendall-Tau correlations on a held-out split mainly demonstrate that the model reproduces the label-generating function. This does not provide independent evidence that REFLEX is better than the baselines at estimating true presentation quality. The claim in Section 6.3.1 that the model 'correlates the highest with the Pseudo Ground Truth' should be reframed as an internal consistency check, and the headline comparison should rest on external validation.","section":"Section 5.2 and Table 1"},{"comment":"The human evaluation, which is the main independent evidence, is limited in two ways. First, it uses only 50 SciDuet presentations with three annotators per sample, and Appendix D instructs annotators that 'Both original and perturbed slides are provided for comparison,' where the perturbed slides are generated by the same metric-specific perturbations used for training. The study therefore partly measures sensitivity to the synthetic corruptions rather than alignment with real-world quality judgments. Second, the paper's abstract claims that REFLEX outperforms state-of-the-art baselines, but Table 3 shows that for Flow, G-Eval is better (rho = 0.36 vs. 0.32; tau = 0.33 vs. 0.29). This discrepancy must be acknowledged, and a realistic transfer test on naturally occurring, non-perturbed presentations is needed to support the central claim.","section":"Section 6.4 and Appendix D, Table 3"},{"comment":"The perturbation-to-quality mapping is assumed rather than validated. The paper sets x = d * 20 and sid = 5 - d for all metrics, but the perturbation functions are not metric-isolating: removing topic slides can also alter flow and redundancy, duplicating slides can alter flow, and permuting images can affect the visual narrative. The linear relationship between perturbation degree and quality is never checked against human judgments. The Limitations section acknowledges model size, non-exhaustive metrics, and lack of design evaluation, but it does not address this validity threat. A calibration study in which human raters score presentations at each perturbation degree, or a justified argument for metric isolation, is necessary before the pseudo-labels can anchor the method's claims.","section":"Section 5.2.1"},{"comment":"The explanation evaluation is also tied to the synthetic training signal. Table 2 computes ROUGE F1 against pseudo-ground-truth explanations produced by the same template-based perturbation functions used to create REFLEX's training targets, so high ROUGE is expected to at least some degree. The human evaluation of explanations in Section 6.4.3 reports only a score distribution and does not provide per-method means, standard deviations, or statistical tests. Please report per-method summary statistics and a significance test to support the claimed superiority in explanation quality.","section":"Section 6.3.3 and Section 6.4.3"}],"minor_comments":[{"comment":"There are several typos and inconsistent notations: 'Actionalble' in Section 1, 'Kendal-Tau' in Section 6.3.1, 'constrative' in the Limitations section, and inconsistent use of 'Reflex' vs. 'REFLEX'. These should be corrected.","section":"Throughout"},{"comment":"The table formatting for 'ROUGE Metric' is misaligned, and the table would benefit from reporting sample sizes and confidence intervals for the ROUGE scores.","section":"Table 2"},{"comment":"The hyperparameter search space is reported, but the selected final hyperparameter values are not. Listing the chosen configuration would improve reproducibility.","section":"Appendix F"},{"comment":"RefSlides is described as a new benchmark dataset, but no availability statement, data release URL, or licensing information is provided. The paper should state how the dataset will be made accessible.","section":"Section 3"},{"comment":"The explanation human evaluation should include a table with per-method mean scores, standard deviations, and inter-annotator agreement by metric, rather than a single aggregate distribution.","section":"Section 6.4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible framework and a potentially useful dataset, but the evaluation strategy is currently too self-referential to support the abstract's claims. The human study is a good start, yet it is small and still partly based on the same synthetic perturbations. I would recommend major revision with a clear request for an external transfer test on naturally occurring presentation quality, corrected treatment of the Flow result, and more complete statistical reporting. I do not think rejection is warranted at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core claim is only half-supported as presented. The automatic correlations in Table 1 are circular, but the human study gives real, if modest, independent signal that REFLEX beats G-Eval on three of four metrics. That is enough to take the work seriously, but not enough to accept the central claim without changes.\n\nWhat is actually new: RefSlides, an 8,111-presentation cross-domain dataset curated from SlideShare, and REFLEX, a reference-free evaluator that fine-tunes Phi3-Mini with LoRA on synthetic negative samples to output per-metric scores and template-based feedback. The perturbation scheme for coverage, redundancy, text-image alignment, and flow is plausible and well described. The human evaluation covers 50 SciDuet decks with three annotators each and reports inter-annotator agreement, which is good practice. REFLEX beats G-Eval on coverage, redundancy, and text-image alignment, though not on flow.\n\nThe soft spots are serious but not fatal. Section 5.2 defines the pseudo-ground-truth score as 5 minus perturbation degree, for the same perturbations used in training, and Table 1 measures correlation against those labels. That is fitting the training target, not independent validation; any baseline that detects the perturbation type would score well. The human study is the only independent check, and it has two weaknesses: only 50 samples, and per Appendix D annotators saw both original and perturbed slides, so they may be rating perturbation degree rather than intrinsic presentation quality. The explanation comparison via ROUGE is weak, since the fine-tuned model is trained to reproduce template strings, and the few-shot baselines are at a structural disadvantage. Flow underperforming G-Eval is buried in Table 3. On the citation pattern: the related work is reasonable for the subfield, and the self-citations to prior Adobe work are contextually appropriate. The paper lists limitations, but not the circularity of the synthetic-label evaluation, which is the load-bearing issue.\n\nWho this is for: people building document-to-slide systems or reference-free multimodal evaluators. RefSlides, if released, would be a solid benchmark. The method is a reasonable baseline, but the evaluation needs rework.\n\nRecommendation: send to peer review, but the authors should be pushed to release the dataset and code, add a transfer test on naturally occurring or differently-perturbed errors, and either expand the human study or reframe the automatic metrics as calibration checks rather than proof of superiority.","headline":"A useful new dataset and a plausible training recipe, but the headline superiority claim rests on a circular automatic evaluation and a small human study that partly rescues it.","tokens_in":16127,"tokens_out":2549,"would_cite":false,"duration_ms":22165,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REFLEX trains an LLM on deliberately broken slides and beats GPT-4o-based scoring for reference-free presentation evaluation.","keywords":["reference-free evaluation","presentation quality metrics","negative sample generation","multimodal evaluation","LLM fine-tuning","actionable feedback","RefSlides benchmark","contrastive learning"],"falsifier":"Collect a set of real presentations with naturally occurring quality variation (e.g., decks from different authors without synthetic corruption), obtain human ratings on the four metrics, and compute the Spearman correlation of REFLEX scores with those ratings; if REFLEX no longer beats the G-Eval baseline on this natural set, the synthetic-perturbation assumption fails.","tokens_in":15203,"feed_emoji":"📊","tokens_out":5947,"duration_ms":44823,"temperature":0.7,"pith_summary":"The paper tries to show that an LLM can judge presentation quality—coverage, redundancy, text-image alignment, and flow—without needing a reference presentation at inference time, and can explain its scores in actionable terms. The trick is to manufacture the training signal: take good human-made decks, corrupt them in controlled metric-specific ways at four intensity levels, assign pseudo scores of 5 minus the degree, and fine-tune a small LLM to produce explanations and scores. The authors build a new benchmark, RefSlides, with 8,111 filtered SlideShare decks, and report that REFLEX beats heuristic embeddings, a same-size prompted LLM, and the GPT-4o-based G-Eval on correlation with pseudo ground truth, with human ratings, and on ROUGE for explanations. If this holds, automatic slide generators can be checked and improved without human references or expensive annotations.","feed_headline":"Deck-grader trained on deliberately broken slides beats GPT-4o","feed_subtitle":"REFLEX scores coverage, redundancy, image-text alignment, and flow with actionable feedback—no reference deck needed.","key_machinery":"The load-bearing object is the perturbation function h^m_d for each metric, which turns positive presentations into negative samples with a pseudo ground-truth score of 5-d and a templated explanation (e.g., removing d*20% of topic-related slides for coverage, duplicating d*20% of slides for redundancy, permuting images across d*20% of slides for alignment, and permuting d*20% of slide positions for flow). These five-way contrastive sets train two LoRA adapters on Phi3-Mini: one generation head for actionable explanations and one single-neuron regression head for scores. The mechanism is that the degree-graded corruptions teach the model to separate fine quality differences and to verbalize what went wrong.","core_discovery":"REFLEX establishes that synthetically generated negative samples—slides with deliberately removed topics, duplicated slides, swapped images, or shuffled orders—carry enough signal to train a reference-free evaluator that outperforms prompt-based LLM evaluators. For each metric, a perturbation function h^m_d maps a good presentation to a negative one of degree d, with pseudo ground-truth score 5-d and a templated explanation. Fine-tuning Phi3-Mini with LoRA on these five-way training sets (one positive, four negative degrees) yields a scoring model and an explanation model that, at inference, take only the presentation (plus the source document for coverage) and return a score in [0,1] and concrete feedback. The central claim is that this contrastive training transfers to real human judgments better than heuristic cosine-similarity metrics or in-context LLM evaluation.","pith_inferences":["Beyond presentations, the same perturbation-and-score recipe could be applied to other structured multimodal outputs—posters, infographics, report layouts—where quality flaws can be synthesized.","The linear score mapping (5-d) is a design choice; a learned mapping from perturbation degree to quality, or probabilistic labels, could improve calibration and is directly testable.","The real-world validity of REFLEX hinges on whether natural presentation flaws resemble the synthetic corruptions; a natural-corruption benchmark with human ratings would settle that.","The method's sensitivity to small perturbation degrees suggests it could double as a diagnostic tool, pinpointing the exact slides responsible for a low score."],"forward_implications":["Presentation generators can be evaluated on the fly, without reference decks, enabling closed-loop refinement of auto-generated slides.","The negative-sample recipe extends to new quality metrics whenever a plausible corruption operator can be defined, making the framework modular.","A small fine-tuned model can beat a much larger prompted LLM on this structured evaluation task, lowering the compute barrier for deployment.","Feedback is generated alongside scores, so users see not just a number but which slides to fix and why.","Human annotation cost for evaluator training is largely replaced by synthetic labeling, though human validation remains necessary."],"supporting_citations":[{"why":"Supplies the SciDuet dataset of document-presentation pairs used for training and evaluation.","marker":"Sun et al., 2021"},{"why":"Provides Phi3-Vision for slide content extraction and Phi3-Mini as the base model for fine-tuning.","marker":"Abdin et al., 2024"},{"why":"LoRA, the parameter-efficient fine-tuning method used to train the adapters.","marker":"Hu et al., 2021"},{"why":"G-Eval, the GPT-4o-based LLM evaluator baseline that REFLEX compares against.","marker":"Liu et al., 2023"},{"why":"Sentence-BERT embeddings used in the heuristic baselines for coverage and redundancy.","marker":"Reimers and Gurevych, 2019"},{"why":"CLIP embeddings underlying the text-image alignment heuristic baseline.","marker":"Radford et al., 2021"}],"fun_headline_variants":["Trained on broken slides, this AI judges decks better than GPT-4o","REFLEX: reference-free deck evaluator trained on negative samples","Broken slides teach AI to grade presentations better than GPT-4","Negative-sample training lifts reference-free presentation scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that each hand-designed perturbation degrades exactly the intended metric, and that the perturbation degree d corresponds linearly to true quality via score = 5 - d, is the load-bearing premise; if real presentation flaws differ from these synthetic corruptions, the trained scores and feedback may not reflect actual quality.","fun_headline_variants_meta":{"raw":{"variants":["Trained on broken slides, this AI judges decks better than GPT-4o","REFLEX: reference-free deck evaluator trained on negative samples","Broken slides teach AI to grade presentations better than GPT-4","Negative-sample training lifts reference-free presentation scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2724,"prompt_tokens":869,"completion_tokens":1855,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1783}},"tokens_in":485,"tokens_out":1855,"duration_ms":12970,"temperature":1.0,"reasoning_tokens":1783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:36:36.025740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of real presentations with naturally occurring quality variation (e.g., decks from different authors without synthetic corruption), obtain human ratings on the four metrics, and compute the Spearman correlation of REFLEX scores with those ratings; if REFLEX no longer beats the G-Eval baseline on this natural set, the synthetic-perturbation assumption fails.","supporting_citations":[],"review_version":1}