{"id":"30aaff8e-4959-408b-a559-7498458912f2","arxiv_id":"2504.14223","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SimplifyMyText is an LLM-powered web tool that rewrites documents into plain language for five audience types, with automatic scores suggesting moderate readability but no human evaluation.","lead":"The authors built a website that uses large language models to rewrite complex texts into plain language for audiences such as students, journalists, and the general public. The site is easy to use, but the paper's evidence that it works comes from automatic readability scores, not from tests with actual readers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 cannot support the audience-specific accessibility claim because the metrics and dataset are not matched to the five target audiences and prompt optimization may reuse the evaluation set.","rationale":"The paper's concrete artifact is real: a public web service with audience selection, file upload, and expert-mode editing, backed by a GitHub repository and demo video. That part of the contribution should be credited. However, the scientific evaluation in Section 3 is the only evidence offered for the effectiveness claim, and it has a fundamental mismatch: PKWP and the automatic metrics are not designed to validate audience-specific plain-language customization. The reader's weakest assumption correctly identifies this. A prompt-optimization step performed on the same corpus would only compound the problem, as the reader also notes. Because the central accessibility claim rests entirely on Table 2, and Table 2 cannot support it without a held-out evaluation and/or human judgment, the CONDITIONAL verdict is appropriate. The paper should be accepted only if the authors add such validation or materially soften the audience-specific and accessibility claims.","tokens_in":5134,"tokens_out":6245,"duration_ms":61887,"concrete_test":"Freeze the exact five audience prompt templates, hold out 1,000 PKWP sentence pairs never used for prompt development, and compute BLEU/SARI/FK against both the original text and a generic zero-shot simplification baseline. Then recruit at least 20 raters per target audience to rate output appropriateness and meaning preservation on a 1-5 scale, comparing system outputs with the generic baseline. If held-out scores drop materially, or if audience-specific outputs are not rated better than generic simple English for their intended group, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence for the claim that GPT-4o and Llama 3.1 produce well-aligned, readable simplifications that enhance accessibility is Table 2, computed on the PKWP dataset. Three properties of this evaluation prevent it from carrying that weight. First, Section 3 states that the three metrics were used both \"to assess performance and optimize prompts\"; no train/test split is described, so the reported scores may be optimistically biased by prompt selection on the same sentences that were later scored. Second, PKWP consists of general Wikipedia/Simple English Wikipedia sentence pairs; it has no references for Scientists, Industry Professionals, Journalists, or other target audiences. Every row of Table 2 is therefore scored against the same generic simple-English references, so BLEU and SARI cannot show that the five audience prompts actually tailor the output. Third, Flesch-Kincaid scores are reported only for the generated text; without the source-text FK scores, the table does not demonstrate that the system improves readability over the input. There is no baseline and no human study. The audience-specific accessibility conclusion is thus unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SimplifyMyText, a web-based demo system that uses LLMs (GPT-4o and Llama 3.1) to generate plain-language simplifications of user-provided text, with five audience-specific prompt options and an expert mode. The authors evaluate the two models on the PWKP dataset (called PKWP in the paper) using BLEU, SARI, and Flesch-Kincaid scores, and report that both models produce well-aligned, readable simplifications. The paper claims to be the first system to offer tailored plain-language simplification for multiple audiences from multiple input formats.","tokens_in":5322,"tokens_out":4790,"duration_ms":37585,"significance":"If the evaluation were sound, the paper would provide a practical, open-source accessible-simplification platform that could benefit researchers and practitioners in accessibility and NLP. The public demo, GitHub repository, and comparison of a proprietary and an open-weights LLM are concrete contributions. However, the current evidence does not establish the central claim of audience-specific accessibility: the metrics and dataset are not matched to the five target audiences, no baselines or human studies are included, and the prompt-optimization procedure is not separated from the evaluation. The system is plausible, but the paper's main quantitative claim needs substantial revision.","major_comments":[{"comment":"Section 3 states that BLEU, Flesch-Kincaid, and SARI are used 'to assess performance and optimize prompts' but provides no information about a train/dev/test split. If the same sentences are used both for prompt engineering and for final scoring, the scores in Table 2 may be optimistically biased and are not independent. This directly affects the conclusion that both models 'generate well-aligned, readable simplifications' and must be addressed by documenting the split or demonstrating that prompt choices were made on a separate development set.","section":"3 Evaluation"},{"comment":"The PKWP dataset (the paper repeats the name PKWP, but the cited source [23] is the PWKP corpus) consists of Wikipedia/Simple Wikipedia sentence pairs with generic simple-English references. It contains no audience-specific references for the five categories in Table 2 (Scientists and Researchers, Students and Academics, Industry Professionals, Journalists and Media, General Public). Therefore BLEU and SARI scores computed against the same generic references cannot show that the audience-specific prompts produce different or appropriate output for each target group; the audience-tailoring claim is not supported by the reported numbers.","section":"3 Evaluation, Table 2"},{"comment":"The Flesch-Kincaid Ease and Grade scores in Table 2 are computed only on the generated text, not on the original inputs, so the table does not demonstrate that simplification improves readability relative to the source. Additionally, there is no baseline (e.g., a generic 'simplify this text' prompt) and no human comprehension test with participants drawn from the target audiences. For a system whose purpose is inclusive plain-language communication, the absence of any human evaluation is a serious gap; the conclusion that accessibility is 'effectively enhanc[ed]' rests on automatic metrics alone, which is insufficient.","section":"3 Evaluation, Table 2"}],"minor_comments":[{"comment":"The sentence 'we https://simplifymytext.org' is missing the verb 'introduce' and is not a complete sentence; please add the missing word.","section":"Abstract"},{"comment":"The metric names should be defined precisely: 'Flesch Reading Ease (FRE)' and 'Flesch-Kincaid Grade Level'; the current 'FK Ease' and 'FK Grade' columns are not introduced in the text.","section":"3 Evaluation, Table 2"},{"comment":"BLEU scores are reported as values between 0.4 and 0.5 while the text says scores range from 0 to 1; BLEU is conventionally reported on a 0-100 scale. Please state the scaling convention used.","section":"3 Evaluation, Table 2"},{"comment":"The parameter count 'approximately 1.8 trillion' for GPT-4o is not officially documented; it should be removed or cited to a reliable source.","section":"3 Evaluation"},{"comment":"The dataset name 'PKWP' in Section 3 and Table 2 should be 'PWKP' to match reference [23] (Zhu et al., 2010); the current name appears to be a typo.","section":"3 Evaluation, Table 2"},{"comment":"The simplified example in Table 1 contains the phrase 'a mistake only 37.5% of the top time and 17.0% of the top five times', which misrepresents 'top-1 and top-5 error rates' and should be corrected or the example replaced with a clearer one.","section":"1 Introduction, Table 1"},{"comment":"The model name appears both as 'Llama 3.1' and 'LLaMA 3.1'; please use one consistent spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is more a demo system description than a full empirical study; if the journal accepts demo/system papers, the framing should be made explicit. The authors should also verify the 'first system' claim, as the paper does not survey existing plain-language tools or LLM-based simplification services, which may make the novelty assertion difficult to defend."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a short demo-system paper. The actual artifact is a live web service (simplifymytext.org) with code on GitHub, supporting typed text and file upload, and five audience-specific prompt presets with an expert mode. That is real and potentially useful, and the writing is clear.\n\nWhat is genuinely new is the product: combining zero-shot LLM simplification with audience-specific plain-language prompts in an interactive tool. The underlying method is the same as Araújo et al. and Swanson et al., and the authors cite them. No new model, no new algorithm, no parameter-free derivation. The \"first system of its kind\" claim is asserted rather than demonstrated; given how fast this space moves, they would need a more careful literature search or a precise definition of \"first.\"\n\nThe soft spot is where the evidence lives. Table 2 reports BLEU, SARI, and Flesch-Kincaid on the PKWP dataset, and Section 3 says the metrics were used both \"to assess performance and optimize prompts.\" No train/test split is described, so the scores may be optimistically biased. More importantly, PKWP contains general Wikipedia/Simple English pairs. It has no references for Scientists, Industry Professionals, Journalists, or the other target audiences. Scoring every audience prompt against the same generic references cannot show tailoring. Flesch-Kincaid is reported only for the generated text, so the table does not show improvement over the source. There is no baseline and no human comprehension test. The stress-test note is right: Table 2 cannot carry the audience-specific accessibility claim.\n\nThat said, this is a demo paper, not a scientific claim paper. The system is plausible, the evaluation is under-powered but not fraudulent. The fix is straightforward: hold out a split, add at least one baseline (e.g., zero-shot generic simplify prompt), report source-text FK, and ideally a small human rating for the audience dimensions. The paper also leaves prompt templates unpublished, which hurts reproducibility.\n\nBottom line: worth a serious referee for a demo track, with the expectation of major revision to the evaluation. I would bring it to a reading group as a case study in how metric choice and dataset mismatch can outrun a system paper's claims.","headline":"A plausible demo system whose audience-specific claim is not supported by the reported evaluation; worth a demo-track referee but not strong enough to cite as evidence.","tokens_in":5861,"tokens_out":1757,"would_cite":false,"duration_ms":15612,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SimplifyMyText is an online system that uses large language models to rewrite complex text into plain language, tailored to a reader group chosen by the user, and the authors report evaluation scores showing readable, well-aligned…","keywords":["text simplification","plain language","large language models","accessibility","audience-specific customization","zero-shot learning","demo system","web platform"],"falsifier":"A comprehension experiment with human participants from each of the five target audiences: if simplified outputs, despite high BLEU and SARI scores and favorable Flesch-Kincaid levels, do not improve factual comprehension over the original texts, the paper's accessibility claim would be unsupported.","tokens_in":4939,"feed_emoji":"📝","tokens_out":6386,"duration_ms":56216,"temperature":0.7,"pith_summary":"The paper introduces SimplifyMyText, an online system that uses large language models to rewrite complex input into plain language, with the target reader selected by the user. The authors describe it as the first such system to accept multiple input formats—typed text, PDFs, and Word documents—and to offer audience-specific customization rather than a one-size-fits-all simplification. They evaluate two models, GPT-4o and Llama 3.1, on a Wikipedia-based simplification dataset and report that both produce outputs with high BLEU and SARI scores and readable Flesch-Kincaid levels across the five offered audience groups. The stated goal is to make tailored plain-language communication easier to produce, thereby supporting literacy and social inclusion.","feed_headline":"Web service simplifies text into plain language per audience","feed_subtitle":"Uses GPT-4o and Llama 3.1 to adapt readability for five target audiences.","key_machinery":"The core mechanism is zero-shot prompting: the user selects one of five audience options (Scientists and Researchers, Students and Academics, Industry Professionals, Journalists and Media, General Public) and the system sends the text to a large language model with instructions to simplify for that audience, defaulting to the General Public setting when no audience is chosen. An expert editing mode lets users click any word for synonyms or definitions and any sentence for rephrasing at different complexity levels, and a rating feature collects feedback. On the evaluation side, the load-bearing tools are the PKWP Wikipedia sentence-pair dataset and the BLEU, SARI, and Flesch-Kincaid metrics, which together quantify alignment with human references and readability.","core_discovery":"The central claim is that an LLM-powered demo system can generate plain-language versions of a text that stay close to human-written references and are readable for a chosen audience, and that the two tested models, GPT-4o and Llama 3.1, both do this effectively. On the PKWP dataset, Llama 3.1 earns the higher BLEU and SARI scores in most user groups, while GPT-4o yields slightly better Flesch-Kincaid Ease and Grade levels; the paper takes these results as evidence that audience-specific plain-language simplification is feasible with zero-shot prompting. It further claims that providing audience selection, word-level synonyms and definitions, sentence-level rephrasing, and a feedback rating feature makes the platform a purpose-built alternative to generic AI text tools.","pith_inferences":["In our reading, the evaluation establishes feasibility, not superiority: the paper does not compare against existing simplification systems or fine-tuned models, so a head-to-head benchmark would be a natural next test.","A direct way to extend the work would be human comprehension testing: ask people from each target audience to read original and simplified versions and answer questions, since BLEU, SARI, and Flesch-Kincaid do not measure understanding directly.","The same platform design could be generalized to non-English plain language, for example German or Spanish, where the paper's own motivation notes large populations with reading challenges; the multi-format upload feature would carry over unchanged."],"forward_implications":["If the system works as described, government agencies could automate plain-language compliance for public documents, reducing the manual labor that currently makes simplification inconsistent.","Educators and content producers could generate audience-appropriate versions of the same material on demand instead of writing separate versions by hand.","Because the architecture accepts any large language model, organizations could plug in open-source or privacy-preserving models rather than relying on a single commercial API.","The built-in rating and feedback loop could steadily improve simplification quality without retraining the underlying model.","The audience-specific options could make the same text usable by diverse readers, extending plain-language practice beyond a single simplified style for everyone."],"supporting_citations":[{"why":"Supplies the PKWP dataset of Wikipedia sentence pairs used for the evaluation.","marker":"[23]"},{"why":"Defines the BLEU metric used to measure alignment between simplified output and human references.","marker":"[16]"},{"why":"Defines the SARI metric used to score simplification quality via added, deleted, and retained words.","marker":"[22]"},{"why":"Establishes the zero-shot and few-shot learning capability that the system's prompting approach relies on.","marker":"[4]"},{"why":"Describes a prior ChatGPT-based simplification approach for non-experts, the closest context for audience-specific output.","marker":"[3]"},{"why":"Shows that fine-tuned large language models can simplify biomedical text, supporting the premise that LLMs are suitable for targeted simplification.","marker":"[20]"}],"fun_headline_variants":["LLM tool rewrites text for five audience levels","Audience-tailored plain language from GPT-4o or Llama 3.1","Demo system simplifies text with zero-shot LLM prompts","SimplifyMyText: custom plain-language output for any reader"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a Wikipedia simplification dataset plus the BLEU, SARI, and Flesch-Kincaid scores are valid proxies for whether the five target audiences actually understand the simplified text.","fun_headline_variants_meta":{"raw":{"variants":["LLM tool rewrites text for five audience levels","Audience-tailored plain language from GPT-4o or Llama 3.1","Demo system simplifies text with zero-shot LLM prompts","SimplifyMyText: custom plain-language output for any reader"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1484,"prompt_tokens":856,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":472,"tokens_out":628,"duration_ms":5639,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:01.085014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A comprehension experiment with human participants from each of the five target audiences: if simplified outputs, despite high BLEU and SARI scores and favorable Flesch-Kincaid levels, do not improve factual comprehension over the original texts, the paper's accessibility claim would be unsupported.","supporting_citations":[{"cited_title":"In: Proceedings of the 23rd International Conference on Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Supplies the PKWP dataset of Wikipedia sentence pairs used for the evaluation."},{"cited_title":"In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Defines the BLEU metric used to measure alignment between simplified output and human references."},{"cited_title":"In: Proceedings of the 33th Annual Conference on Neural Informa- tion Processing Systems","cited_arxiv_id":null,"evidence_quote":"Establishes the zero-shot and few-shot learning capability that the system's prompting approach relies on."},{"cited_title":"In: Perspectives and Trends in Education and Tech- nology, Smart Innovation, Systems and Technologies, vol","cited_arxiv_id":null,"evidence_quote":"Describes a prior ChatGPT-based simplification approach for non-experts, the closest context for audience-specific output."}],"review_version":1}