Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation

Faezeh Hosseini; Sara Rezaeimanesh; Yadollah Yaghoobzadeh

REVIEW 4 major objections 5 minor 2 references

Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Claude-3.5-Sonnet achieves the highest idiom-translation accuracy in both English-to-Persian and Persian-to-English in this comparison of LLMs, NMT systems, and hybrids.

desk verdict A valuable Persian idiom resource with a plausible but under-supported Claude ranking, because the GPT-4o judge is validated on only a fraction of the evaluated systems. read the letter →

arxiv 2412.09993 v2 pith:SV64R5KD submitted 2024-12-13 cs.CL

Sara Rezaeimanesh , Faezeh Hosseini , Yadollah Yaghoobzadeh This is my paper · ORCID

classification cs.CL

keywords idiomtranslationPersianEnglishlargelanguagemodelsmachineLLM-as-a-judgepromptengineeringparalleldatasets

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the first parallel Persian–English datasets for idiom translation—200 sentences in each direction, with Persian idioms drawn from a new PersianIdioms resource of 2,200 idioms—and uses them to compare open and closed-source LLMs, NMT models, and hybrid LLM-then-NMT pipelines under several prompting strategies. Its central empirical claim is that Claude-3.5-Sonnet achieves the highest idiom-translation accuracy in both directions, reaching GPT-4o-judged scores of 94.0 for English-to-Persian and 75.0 for Persian-to-English. The paper also establishes that models translate English idioms far more accurately than Persian ones, that weaker LLMs improve in English-to-Persian when combined with Google Translate, and that the best prompting style depends on model strength and direction. If the results are correct, they provide a concrete, reproducible benchmark for idiom translation in this language pair and a validated evaluation recipe for future work.

What carries the argument

The central objects are the two new parallel datasets (Fa→En and En→Fa, each 200 sentences with one idiom per sentence) and the evaluation protocol built around them: a binary idiom-translation metric defined within the MQM framework, fluency ratings on a 1–5 scale, and GPT-4o-as-judge with reference-guided grading, which the paper validates by Spearman correlation against human scores. The other load-bearing mechanism is the hybrid pipeline, in which an LLM first identifies idioms and replaces them with literal clauses and an NMT system then translates the resulting text; this pipeline is what produces the finding that weaker LLMs gain from combination with Google Translate in English→Persian.

What would settle it

Have multiple native Persian and English speakers annotate idiom-translation correctness for all 200 sentences across every model and prompt in Table 6, then rank systems by human majority vote and compare with the GPT-4o ranking; if the Spearman correlation drops below roughly 0.8 or the top system changes, the paper's central ranking is falsified.

Watch

Extended reading notes

Core claim

On two new parallel test sets of 200 sentences each, Claude-3.5-Sonnet obtains the highest GPT-4o idiom-accuracy scores in both translation directions among all tested systems: 94.0 for English→Persian with the chain-of-thought prompt and 75.0 for Persian→English with the multi-prompt setup, where GPT-4o scores are the paper's primary idiom-translation metric validated against human labels. The paper further claims that combining a weaker LLM with Google Translate significantly improves English→Persian idiom translation—raising Qwen-2.5-72B's GPT-4o score from 74.5 to 88.0 and GPT-3.5's from 72.0 to 79.0—while Persian→English outputs benefit from simple single prompts for GPT-3.5, GPT-4o-mini, and Qwen-2.5, and from complex CoT or MultiPrompt setups for Claude-3.5-Sonnet and Command R+. These findings are supported by manual annotation of 100 sentences from seven model outputs and by GPT-4o-as-judge scores on the full 200 sentences per direction.

Load-bearing premise

The ranking of all models rests on GPT-4o-as-judge scores, but the judge's agreement with human annotators was tested on only seven outputs—all from GPT-3.5, Google Translate, or their combination—and on the first 100 sentences of those outputs; the paper assumes this reliability transfers to Claude, Qwen, Command R+, NLLB, and all prompt variants without direct validation.

Editorial extensions

If this is right

Claude-3.5-Sonnet becomes the default strong baseline for future Persian–English idiom-translation research, since it tops both directions on the paper's metrics.
For English→Persian, teams with limited compute can approximate strong-LLM performance by chaining an open-weight LLM such as Qwen-2.5-72B in front of Google Translate.
For Persian→English, prompt choice should be matched to model strength: single prompts for weaker models, chain-of-thought or multi-step prompts for larger ones.
The PersianIdioms resource and the two parallel datasets give the community a way to measure progress on a lower-resource figurative-language task that previously had no benchmark.
GPT-4o-as-judge, BLEU, and BERTScore can be used as a substitute for manual idiom-translation and fluency evaluation when ranking systems, provided the judge is re-validated on the target model outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If GPT-4o-as-judge reliability holds beyond the seven validation outputs, the same reference-guided binary-judge protocol could be exported to other low-resource language pairs, but the paper's own note that GPT-4o tends to underestimate paraphrased English→Persian translations suggests the judge may need reference-set expansion before ranking is trustworthy.
The hybrid gain pattern—an idiom-aware rewriter plus a fluent NMT backend—suggests a general complementarity principle that could be tested on other language pairs and other figurative-language phenomena such as metaphors and proverbs.
The 200-sentence sample and single-judge automation leave room for a stress test: re-annotating all outputs with multiple native-speaker judges could reveal whether the Claude-vs-GPT-4o-mini margins in Table 6 are stable or within annotation noise.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The main thing this paper contributes is PersianIdioms: 2,200 Persian idioms with meanings, 700 with examples, plus two 200-sentence parallel idiom translation test sets with expert-validated gold translations. That is a real resource gap, and the paper is honest about its construction process. I'm not aware of another Persian idiom resource of this size, and the parallel sets are useful for future MT evaluation.

What the paper does well beyond the resource: it runs a broad comparison of open and closed LLMs, NMT models, prompts, and LLM+NMT hybrids, and it reports manual evaluation with inter-annotator agreement on a subset. The manual scoring procedure is explained clearly, and the choice of Gwet's AC1 for skewed fluency scores is reasonable.

The soft spot is the one the stress-test flags, and I think it lands. GPT-4o is used as the primary idiom-accuracy judge for Table 6, but it is validated only on seven model outputs, all from GPT-3.5, Google Translate, or their combination, and on the first 100 sentences. The paper even says GPT-4o tends to underestimate paraphrastic translations, which is exactly what Claude's CoT and MultiPrompt outputs are most likely to be. The n=7 Spearman correlations are suggestive but not enough to establish per-system calibration. So the headline 'Claude is best in both directions' is plausible but not proven. The absence of error bars or significance tests on the Table 6 differences (e.g., 94.0 vs 91.0) compounds that. I'd like to see the judge validated on a sample of Claude and GPT-4o-mini outputs, or at least a sentence-level human evaluation on the systems that matter for the ranking.

Minor notes: the dataset size is acknowledged in the limitations; the hybrid method is adapted from prior work, but that's a legitimate extension to a new language pair. The paper's tone is appropriately cautious except for the word 'outstanding' in the abstract and conclusion, which overstates what the evidence supports.

Who this is for: anyone working on Persian MT, idiom translation, or LLM-as-judge reliability. I'd bring it to a reading group as an example of a resource paper with a somewhat fragile evaluation. It deserves peer review, but I'd want the judge-validation issue addressed before publication, or at least clearly acknowledged as a limitation. My verdict: send it to review, with a request for additional validation.

Referee Report

4 major / 5 minor

Summary. The paper contributes a new Persian idiom resource (PersianIdioms, 2,200 idioms, 700 with examples) and two parallel evaluation sets of 200 sentences each for English→Persian and Persian→English idiom translation, drawing English sentences from EPIE and MAGPIE and Persian sentences from PersianIdioms. It then evaluates five LLMs (GPT-3.5-turbo, GPT-4o-mini, Qwen-2.5-72B, Command R+, Claude-3.5-Sonnet), three NMT systems (NLLB-200, MADLAD-400, Google Translate), and hybrid LLM+NMT combinations under several prompting schemes (three single prompts, a chain-of-thought prompt, and a multi-step prompt). Translation quality is assessed with manual scores for idiom accuracy and fluency on a subset of outputs and with automatic metrics (COMET, BERTScore, BLEU, GPT-4o-as-judge). The central empirical claim is that Claude-3.5-Sonnet achieves the best GPT-4o idiom-accuracy scores in both translation directions, and that weak LLMs improve when combined with Google Translate in English→Persian, while Persian→English translation favors simpler prompts for weaker models and complex prompts for stronger ones.

Significance. The datasets and the PersianIdioms resource are potentially useful contributions to a low-resource language pair, and the paper is one of the few to compare prompting methods and LLM+NMT hybrids for idiom translation in Persian. The authors are transparent about data sources, manual annotation procedures, and inter-annotator agreement, and they release the data, which supports reproducibility. The comparison of automatic metrics against manual scores is a useful sanity check. However, the strength of the headline ranking is contingent on the validity of the GPT-4o judge across all evaluated systems; that validity is currently established only on a small, system-restricted sample. Consequently, the paper's contribution is significant conditional on the additional validation requested below.

major comments (4)

[§5.2, §5.3, §4.3, Table 4 and Table 6] The GPT-4o-as-judge is calibrated only against manual labels on seven outputs, all from GPT-3.5, Google Translate, or their combination, and only on the first 100 of 200 sentences. The paper then uses GPT-4o scores in Table 6 as the primary idiom-accuracy metric for every system and prompt, including Claude-3.5-Sonnet, Qwen-2.5, Command R+, GPT-4o-mini, NLLB, and MADLAD. Because §4.3 reports that GPT-4o tends to underlabel flexible paraphrases—precisely the kind of output Claude's CoT and MultiPrompt settings are designed to produce—the headline En→Fa gap between Claude (94.0) and GPT-4o-mini (91.0), and the Fa→En ranking, could be a judge artifact. Please add manual idiom-accuracy scores (or a substantial validation sample) for all models that are ranked, and report per-system judge agreement.
[§5.3, Table 4] The Spearman correlations used to establish the reliability of GPT-4o (0.88 and 0.79) are computed over n=7 aggregated model outputs, with no confidence intervals, significance tests, or scatterplots. With n=7, the rank correlation is highly sensitive to a single output, and the statement "GPT-4o performs comparably to humans" is stronger than the evidence supports. Please report bootstrap or permutation intervals, or otherwise quantify the uncertainty around these correlations.
[§5.4, §5.5, Table 6] Table 6 gives point estimates without error bars, and Section 5.1 only states that GPT outputs were run "multiple times" without reporting the number of runs or the spread. Many of the comparisons in the GPT-4o column differ by only 1–3 points (for example, En→Fa Claude SinglePrompt2=93.0 vs. SinglePrompt3=93.5, or GPT-4o-mini SinglePrompt2=90.0 vs. MultiPrompt=91.0), so claims of superiority need variance estimates or significance tests. Please provide repeated-run statistics (mean ± std) or significance tests, or explicitly describe the results as exploratory.
[§7 Limitations] The limitations section acknowledges dataset size and language coverage but not the judge-transfer limitation described above. Because the central claim relies on GPT-4o scores for systems that were never manually evaluated, this missing limitation should be stated explicitly and discussed.

minor comments (5)

[§3.2 / References] MAGPIE is cited as (Xu et al., 2024), but that reference is a paper on alignment data synthesis, not the MAGPIE idiom corpus; the En→Fa dataset's provenance therefore needs a correct citation (Haagsma et al., 2020).
[§5.1] The choice of temperature 0.8 for translation and 0.1 for the judge is not justified; please add a sentence explaining the rationale.
[Table 6] Row labels such as "rSinglePrompt1" and the lack of clear grouping make the table difficult to read; please reformat so that model, prompt, and metric columns are unambiguous.
[Table 7] The source/back-translation uses "Poor Mrs" and then "Poor mother"; please clarify the source word and make the glosses consistent.
[Abstract / §5.3] The abstract states that BLEU and BERTScore are "effective" for evaluation, but Table 4 shows their correlation with idiom-translation scores is near zero (-0.03 and 0.18–0.25); please specify that they are useful for fluency only.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical evaluation in which system outputs are generated independently of the evaluation metrics, and the GPT-4o judge is checked against manual labels rather than defined in terms of the systems it ranks.

full rationale

This is an empirical benchmarking paper, not a derivation. The central claim that Claude-3.5-Sonnet performs best in both translation directions (Sections 5.4 and 5.5, Table 6) follows from automatic scores computed on independently generated translations, so there is no fitted parameter being renamed as a prediction and no equation that reduces to its own input. The LLM-as-a-judge step is the only potentially load-bearing methodological choice: GPT-4o scores are used to rank idiom translation accuracy after being validated against manual annotations on seven model outputs (Section 5.3). That validation is small and is confined to GPT-3.5, Google Translate, and their combinations, which could threaten the robustness of the ranking if GPT-4o's judgments degrade on Claude or other outputs. However, this is a concern about external validity and evaluation robustness, not circularity: the judge's labels are not constructed from the ranking it is used to produce, and the paper explicitly reports the correlation and agreement values rather than assuming the judge is correct by definition. The PersianIdioms resource and the parallel datasets are newly introduced in the paper, not imported from prior self-citation chains, and no uniqueness or ansatz argument is invoked to force a result. The limitation about dataset size is acknowledged in Section 7, but even without that acknowledgment, the evaluation design does not make any claim equivalent to its inputs by construction. Therefore no circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's conclusions rest on empirical assumptions about dataset representativeness and evaluation reliability rather than on fitted parameters or invented entities. The hand-chosen temperatures are the only numeric settings that influence all results.

free parameters (2)

LLM generation temperature = 0.8
Hand-chosen in Section 5.1 to reduce response variability while preserving creative freedom. All LLM outputs and subsequent rankings are conditional on this setting.
GPT-4o judge temperature = 0.1
Set to minimize response variation for the LLM-as-a-judge evaluation (Section 4.3).

assumptions (4)

domain assumption The 100-sentence manual evaluation subset is representative of the full 200-sentence datasets.
Manual scoring is performed only on the first 100 sentences from seven outputs (Section 5.2); correlations and agreement scores are based on this subset.
domain assumption GPT-4o judge scores generalize to all models and prompts.
GPT-4o is validated on seven outputs from GPT-3.5, Google Translate, and their combination (Section 5.3), then used for all other models including Claude, Qwen, and Command R+.
domain assumption The curated idioms are representative of Persian and English idiom difficulty.
Fa-En idioms were filtered by Google search popularity and manual exclusions; En-Fa sentences were manually selected from EPIE and MAGPIE (Section 3.2).
domain assumption PersianIdioms resource is accurate.
Native speakers reviewed the resource, but the verification process is not described in detail (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation." pith.science (2026). https://pith.science/paper/SV64R5KD

@misc{pith2026241209993,
  author       = {Pith},
  title        = {Pith review of: Large Language Models for Persian $ \leftrightarrow $ English Idiom Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SV64R5KD}},
  note         = {Machine review of arXiv:2412.09993}
}

abstract

Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems. However, the impact of different prompting methods and LLM-NMT combinations on idiom translation has yet to be thoroughly investigated. This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian$\rightarrow$English and English$\rightarrow$Persian translations, with Persian idioms sampled from our PersianIdioms resource, a collection of 2,200 idioms and their meanings, with 700 including usage examples. Using these datasets, we evaluate various open- and closed-source LLMs, NMT models, and their combinations. Translation quality is assessed through idiom translation accuracy and fluency. We also find that automatic evaluation methods like LLM-as-a-judge, BLEU, and BERTScore are effective for comparing different aspects of model performance. Our experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions. For English$\rightarrow$Persian, combining weaker LLMs with Google Translate improves results, while Persian$\rightarrow$English translations benefit from single prompts for simpler models and complex prompts for advanced ones.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[2014]

Yasmin Moslem, Rejwanul Haque, John D

Multidimensional quality metrics (mqm): A frameworkfordeclaringanddescribingtranslation qualitymetrics. Yasmin Moslem, Rejwanul Haque, John D. Kelle her, and Andy Way. 2023. Adaptive machine translation with large language models. Preprint, arXiv:2301.13294. Makoto Nagao. 1984. Aframework of a mechanical translationbetweenjapaneseandenglishbyanalogy prin...

arXiv 2023
[2024]

Preprint,arXiv:2410.00988

Creativeandcontextawaretranslationofeast asianidiomswithgpt4 . Preprint,arXiv:2410.00988. NLLBTeam,MartaR.Costajussà,JamesCross,Onur Çelebi,MahaElbayad,KennethHeafield,KevinHef fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard,Anna Sun, SkylerWang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar rault, Gabriel Mejia Gonzalez, Pran...

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2014]

Yasmin Moslem, Rejwanul Haque, John D

Multidimensional quality metrics (mqm): A frameworkfordeclaringanddescribingtranslation qualitymetrics. Yasmin Moslem, Rejwanul Haque, John D. Kelle her, and Andy Way. 2023. Adaptive machine translation with large language models. Preprint, arXiv:2301.13294. Makoto Nagao. 1984. Aframework of a mechanical translationbetweenjapaneseandenglishbyanalogy prin...

arXiv 2023

[2] [2024]

Preprint,arXiv:2410.00988

Creativeandcontextawaretranslationofeast asianidiomswithgpt4 . Preprint,arXiv:2410.00988. NLLBTeam,MartaR.Costajussà,JamesCross,Onur Çelebi,MahaElbayad,KennethHeafield,KevinHef fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard,Anna Sun, SkylerWang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar rault, Gabriel Mejia Gonzalez, Pran...

arXiv