Translating Classical Poetry into Modern Prose

Chalamalasetti Kranti; Sowmya Vajjala

arxiv: 2606.02806 · v1 · pith:XGLVOYZJ · submitted 2026-06-01 · cs.CL

Translating Classical Poetry into Modern Prose

Chalamalasetti Kranti , Sowmya Vajjala This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 14:30 UTCgrok-4.3pith:XGLVOYZJ record.json open to challenge →

classification cs.CL

keywords Telugu poetrypoem to prose translationLLM evaluationmachine translation datasetclassical literaturelow-resource translation

0 comments

The pith

The paper presents a dataset for translating classical Telugu poetry to modern prose and demonstrates that current LLMs have substantial room for improvement on this task.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This work creates Padyam2Gadyam, a collection of 600 13th-17th century Telugu poems with verified prose translations in Telugu and English. Five LLMs are tested on generating these translations, revealing performance differences but overall significant shortcomings in both languages. The findings matter because they provide a new benchmark for literary machine translation in a low-resource setting and show where standard evaluation methods fall short for poetry. Qualitative discussion points to specific capabilities and limits of existing approaches.

Core claim

The authors establish that the Padyam2Gadyam dataset allows systematic evaluation of poem-to-prose translation, and that contemporary LLMs, despite varying performance, leave a large room for improvement in producing accurate modern prose from classical Telugu poetry in both target languages.

What carries the argument

The Padyam2Gadyam dataset of 600 poems with human-verified prose translations, serving as a benchmark to assess LLM translation quality.

Load-bearing premise

The human-verified Telugu and English prose translations accurately represent the intended meaning of the classical poems and can serve as reliable gold-standard references for measuring LLM output quality.

What would settle it

Demonstrating that LLMs can generate translations that human evaluators rate as equivalent in meaning to the gold standards on a majority of the poems would falsify the claim of large room for improvement.

Figures

Figures reproduced from arXiv: 2606.02806 by Chalamalasetti Kranti, Sowmya Vajjala.

**Figure 1.** Figure 1: Overview of our task. Given a classical Telugu poem, LLMs generate Telugu and English prose translations. Outputs generated by Gemini-3.1-Flash-Lite model are seen here. The scatter plots compare reference-based COMET and normalized LLM-judge scores across models for TE–TE and TE–EN translation settings [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Figure 2: Dataset Example Dataset Construction: The poems were sourced from publicly available online archives, including Archive.org 3 and Telugu Wikisource 4 , from books that have the poems along with its meaning in Telugu prose. We include a poem in the dataset only when both the poem and its Telugu prose version are available from the same source. To include poems from different time periods, we shortlist poe… view at source ↗

read the original abstract

We introduce Padyam2Gadyam, a dataset for the task of poem-to-prose translation from 13th-17th Century Telugu Classical Poetry to contemporary Telugu and English prose. The dataset consists of 600 poems and their human-verified Telugu and English prose translations. We evaluated 5 contemporary Large Language Models (LLMs) on their ability to do poem-to-prose translation into Telugu and English. Our results indicate that while there are differences across LLMs, their overall performance leave a large room for improvement in both languages. Through qualitative analysis, we discuss the the capabilities and limitations of contemporary MT evaluation approaches for this task.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

New dataset for classical Telugu poem-to-prose translation with basic LLM tests, but the human verification process lacks needed detail on agreement and expertise.

read the letter

The main takeaway is a new dataset called Padyam2Gadyam with 600 classical Telugu poems and their human-verified prose versions in both Telugu and English. The authors test five LLMs on the task and note that performance still has substantial gaps in both target languages, while also flagging that standard MT metrics do not fit this kind of work well.

The dataset itself is the clearest addition. No prior published resource matches this exact poem-to-prose setup for 13th-17th century Telugu, so the resource fills a narrow but real gap for people working on Telugu literature or low-resource literary translation. Running the LLMs and including a qualitative look at metric shortcomings is straightforward and useful as a starting point.

The soft spot is the reference quality. The abstract calls the translations human-verified, yet supplies no numbers on how many verifiers handled each item, what their agreement was, or how they resolved differences. Classical poetry often allows multiple valid readings, so single-pass or low-agreement verification could introduce noise that directly affects the measured performance gap. That detail matters more than the authors appear to have treated it.

The paper stays within its stated bounds and does not overclaim broader impact. It is aimed at researchers building MT resources for South Asian languages or evaluating LLMs on creative text. A reader focused on dataset papers or narrow-domain benchmarks would get concrete value from the resource and the initial numbers.

It deserves peer review. The dataset is new and the evaluation is reproducible enough to be worth referee time, provided the annotation process is documented more fully in revision.

Referee Report

2 major / 2 minor

Summary. The paper introduces Padyam2Gadyam, a new dataset of 600 13th-17th century Telugu classical poems paired with human-verified prose translations in both contemporary Telugu and English. It benchmarks five contemporary LLMs on the poem-to-prose translation task in both target languages and concludes that performance differences exist across models but that substantial room for improvement remains; a qualitative discussion addresses limitations of standard MT metrics for this genre.

Significance. If the gold-standard references prove reliable, the dataset would constitute a useful resource for low-resource-language poetry translation research, an area with few existing parallel corpora. The empirical LLM evaluation and metric-limitation analysis would provide a concrete starting point for future work on culturally and stylistically complex translation.

major comments (2)

[Dataset section] Dataset section: The claim that the 600 Telugu and English prose translations are 'human-verified' gold standards is load-bearing for every quantitative and qualitative result, yet the manuscript provides no information on the number of verifiers, their linguistic expertise, inter-annotator agreement, or disagreement-resolution procedure. Classical Telugu poetry routinely admits multiple valid interpretations; without these details the measured performance gap and the 'large room for improvement' conclusion cannot be assessed.
[Evaluation / Results section] Evaluation / Results section: The headline finding that 'overall performance leave a large room for improvement' is asserted without reported numerical scores (e.g., BLEU, chrF, or human judgments), confidence intervals, or statistical tests comparing the five LLMs against each other or against any baseline. The abstract alone does not allow verification of effect sizes or whether the gap is practically meaningful.

minor comments (2)

[Abstract] Abstract: duplicated definite article ('discuss the the capabilities').
[Dataset section] The manuscript should clarify whether the English and Telugu prose references were produced independently or via translation of one another, as this affects the independence of the two evaluation tracks.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and will make the necessary revisions to strengthen the manuscript.

read point-by-point responses

Referee: [Dataset section] Dataset section: The claim that the 600 Telugu and English prose translations are 'human-verified' gold standards is load-bearing for every quantitative and qualitative result, yet the manuscript provides no information on the number of verifiers, their linguistic expertise, inter-annotator agreement, or disagreement-resolution procedure. Classical Telugu poetry routinely admits multiple valid interpretations; without these details the measured performance gap and the 'large room for improvement' conclusion cannot be assessed.

Authors: We agree that the verification details are essential for establishing the dataset as reliable gold standards, particularly given the interpretive flexibility of classical Telugu poetry. The manuscript does not currently include these specifics beyond the 'human-verified' descriptor. We will revise the Dataset section to provide a complete description of the verification process, including the number of verifiers, their expertise, inter-annotator agreement, and disagreement-resolution procedure. This will allow readers to properly evaluate the results. revision: yes
Referee: [Evaluation / Results section] Evaluation / Results section: The headline finding that 'overall performance leave a large room for improvement' is asserted without reported numerical scores (e.g., BLEU, chrF, or human judgments), confidence intervals, or statistical tests comparing the five LLMs against each other or against any baseline. The abstract alone does not allow verification of effect sizes or whether the gap is practically meaningful.

Authors: We acknowledge that the abstract does not report the specific numerical scores, making it difficult to assess the claims directly from the abstract. While the Results section presents the model evaluations, we agree that adding confidence intervals, statistical tests, and a summary of key metrics (such as BLEU and chrF) to the abstract would improve verifiability and demonstrate the practical significance of the gaps. We will revise the abstract and enhance the Results section accordingly. revision: yes

Circularity Check

0 steps flagged

No circularity: dataset creation plus empirical LLM evaluation

full rationale

The paper introduces the Padyam2Gadyam dataset of 600 poems with human-verified Telugu and English prose translations, then reports LLM performance on poem-to-prose translation into both languages. No equations, fitted parameters, predictions derived from subsets of the same data, or self-citations appear in the provided text. The central claim (LLMs leave large room for improvement) is an empirical measurement against the created references and does not reduce to any input by construction. This matches the default case of a self-contained empirical contribution with no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical dataset-creation and benchmarking paper with no mathematical derivations or modeling assumptions that introduce free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5628 in / 1095 out tokens · 33202 ms · 2026-06-28T14:30:35.206253+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages · 1 internal anchor

[1]

No Language Left Behind: Scaling Human-Centered Machine Translation

ijcai.org. Costa-Jussà, M. R., Cross, J., Çelebi, O., El- bayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022). No language left behind: Scaling human- centered machine translation. arXiv preprint arXiv:2207.04672. Das, K. K., Jagadeeshan, M. B., Sahith, N. C., Sandhan, J., and Goyal, P. (2025). Still not t...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[2]

Present an optimistic message about improvement rather than a warning about malevolent nature 4

Introduce religious/devotional elements (Bhaskara worship) that may not be the primary focus 3. Present an optimistic message about improvement rather than a warning about malevolent nature 4. Diverge substantially from the concise moral lesson in the reference The grammar and prose quality of the machine output is good, but the seman- tic preservation is...
[3]

Minor: ”pest” vs the original’s more general concept might be slightly narrower Overall assessment: The translation preserves the meaning very well

The rhetorical question format, while poetic, may be slightly interpretive beyond the source 3. Minor: ”pest” vs the original’s more general concept might be slightly narrower Overall assessment: The translation preserves the meaning very well. The core philosophical point about wicked people being unable to do good, paral- leled with a pest’s inability t...

[1] [1]

No Language Left Behind: Scaling Human-Centered Machine Translation

ijcai.org. Costa-Jussà, M. R., Cross, J., Çelebi, O., El- bayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022). No language left behind: Scaling human- centered machine translation. arXiv preprint arXiv:2207.04672. Das, K. K., Jagadeeshan, M. B., Sahith, N. C., Sandhan, J., and Goyal, P. (2025). Still not t...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[2] [2]

Present an optimistic message about improvement rather than a warning about malevolent nature 4

Introduce religious/devotional elements (Bhaskara worship) that may not be the primary focus 3. Present an optimistic message about improvement rather than a warning about malevolent nature 4. Diverge substantially from the concise moral lesson in the reference The grammar and prose quality of the machine output is good, but the seman- tic preservation is...

[3] [3]

Minor: ”pest” vs the original’s more general concept might be slightly narrower Overall assessment: The translation preserves the meaning very well

The rhetorical question format, while poetic, may be slightly interpretive beyond the source 3. Minor: ”pest” vs the original’s more general concept might be slightly narrower Overall assessment: The translation preserves the meaning very well. The core philosophical point about wicked people being unable to do good, paral- leled with a pest’s inability t...