REVIEW 3 cited by
BLEU is Not Suitable for the Evaluation of Text Simplification
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects. In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation. We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments. We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved. Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.
Forward citations
Cited by 3 Pith papers
-
Building Large-Scale English-Romanian Literary Translation Resources with Open Models
LoRA-fine-tuning a 12B open model on 15,000 GPT-o3 fable translations yields rubric scores close to GPT-o3 (4.83 vs 4.92) for English-to-Romanian literary translation at roughly one percent of the API cost.
-
The MedPerturb Dataset: What Non-Content Perturbations Reveal About Human and Clinical LLM Decision Making
MedPerturb finds that LLMs are more sensitive to gender and style changes in clinical text, while medical students are more sensitive to LLM-generated summaries and dialogues, in triage decisions.
-
Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation
Human-written plain language summaries led to significantly better reader comprehension than LLM-generated ones, despite similar subjective ratings, and most automated metrics did not predict comprehension.
Discussion (0). Continue with ORCID to comment.