MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language

Amirhossein Shabani; Amirmohammad Salehoof; Farbod Bijary; Farhan Farsi; Farnaz Aghababaloo; Ghazal Zamaninejad; MohammadAli SadraeiJavaheri; Parsa Ghofrani; Saeedeh Momtazi; Shahriar Shariati Motlagh

REVIEW 4 major objections 6 minor 6 references

MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces MELAC, a 19-dataset benchmark for Persian language and Iranian culture, and reports that only one of 41 current LLMs exceeds 50% accuracy on Iranian-specific content, evidence of a genuine cultural-alignment gap.

desk verdict Useful Persian benchmark with a headline cultural-gap claim that rests on two small private legal/religious datasets and needs to be reined in. read the letter →

arxiv 2508.00673 v1 pith:KOG76LWS submitted 2025-08-01 cs.CL

Farhan Farsi , Farnaz Aghababaloo , Shahriar Shariati Motlagh , Parsa Ghofrani , MohammadAli SadraeiJavaheri , Shayan Bali , Amirhossein Shabani , Farbod Bijary

show 3 more authors

Ghazal Zamaninejad AmirMohammad Salehoof Saeedeh Momtazi

This is my paper

classification cs.CL

keywords PersianlanguageIraniancultureculturalalignmentLLMbenchmarkevaluationdatasetmultiple-choicedatacontaminationNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to measure whether large language models actually understand Iranian culture and Persian linguistic structure, rather than merely translating English. It introduces MELAC, a suite of 19 evaluation datasets: 13 newly authored and 6 translated-and-localized versions of established benchmarks, covering Persian grammar, homographs, proverbs, literature, Iranian law, religious rules, university entrance exams, reading comprehension, and common-sense reasoning. The authors evaluated 41 open and closed models on private test sets to reduce the risk of data contamination. The central result is that all models perform poorly on Iranian-specific content: only one of the 41 exceeds 50% accuracy, and even Persian-fine-tuned models lag clearly on Persian-specific tasks. If correct, this means current LLMs are far less culturally aligned with Iran than their general English benchmarks suggest.

What carries the argument

The central object is the MELAC benchmark suite: 19 evaluation datasets, each split into private test items and designed to probe a distinct slice of Persian language or Iranian culture, including grammar, homographs, proverbs, literature, law, religious practice, entrance exams, reading comprehension, and common-sense reasoning. Its load-bearing design choices are (1) thirteen newly authored datasets with expert review rather than translated content, so correct answers depend on local knowledge; (2) six translated-and-localized versions of established English benchmarks; and (3) a private held-out evaluation protocol using log-probability scoring for multiple-choice items and exact-match or F1 for generative items, which the paper argues limits the inflation that public benchmark contamination causes.

What would settle it

Independently re-answer a random sample of Expert-Eval, Iran-Law, Religion-Rules, and Proverbs-Quiz items by two native Persian-speaking experts who have not seen MELAC's labels, then compare agreement. If the experts disagree with the released gold answers on a substantial share of items, or if a rerun of the same 41 models on the audited sample puts more than one model above 50% accuracy, the paper's central cultural-gap claim would be called into question.

Watch

Extended reading notes

Core claim

The paper's central claim is that there is a real and measurable cultural-alignment gap between current LLMs and Persian-Iranian knowledge. The 19 MELAC datasets are designed so that translation alone cannot inflate scores: six are localized versions of English benchmarks, and thirteen are new items built from Persian curricula, legal and religious content, homographs, proverbs, and transcribed Iranian examinations. On the Persian-specific portion of the benchmark, the paper reports that only one of the 41 models achieves over 50% accuracy, and Persian-fine-tuned models such as Maral, PersianMind, and Dorna do not outperform general multilingual models. The paper interprets this as evidence that current models have not absorbed the cultural knowledge needed for reliable use in Persian-speaking contexts.

Load-bearing premise

The benchmark's scores are only as trustworthy as its reference answers: the manually transcribed Expert-Eval exam items and the internally generated reading-comprehension passages must be correct and unambiguous, and because the test sets are private no outside team can verify them independently.

Editorial extensions

If this is right

Iranian-specific applications such as legal advice, religious guidance, and Persian literary instruction cannot yet be delegated to an LLM without human review.
Translation-based evaluation overstates cultural ability: models that handle Persian text still miss idiomatic, legal, and religious content that depends on Iranian context.
Fine-tuning on Persian text does not by itself confer Iranian cultural competence, since the Persian-tuned models in this study trail general multilingual models.
Private, refreshed test sets are a necessary guard against benchmark contamination for cultural evaluation.
Closed API models lead the Persian rankings overall, while open and Persian-tuned models are clustered near the bottom on culturally specific tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If MELAC's test sets remain private and versioned, later evaluation runs could be used to detect contamination: a sudden jump in scores after a public release of benchmark items would indicate leakage rather than genuine cultural learning.
The same dataset-building pattern of expert-authored original items plus localized standard benchmarks could be transplanted to other under-represented Persian-speaking communities, such as Afghanistan and Tajikistan, to test whether Persian alignment generalizes beyond Iran.
A controlled fine-tuning experiment that adds Iranian cultural documents, such as law, religion, and literature, to an open model while holding general Persian text constant would isolate whether cultural content or language volume drives the observed gap.
If the 50-percent threshold persists after independent expert auditing, the practical implication is that frontier training pipelines need curated non-English cultural corpora, not only larger multilingual text mixtures.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a solid, workmanlike benchmark contribution: 13 new Persian and Iranian-culture datasets, 6 localized adaptations, 41 models evaluated with a standard lm-evaluation-harness protocol, and a reasonable attempt at contamination control via private test sets. The translation quality checks with human plus GPT-4 scores are a nice touch, and the option-format control is the kind of detail that makes the eval credible. For anyone working on Persian NLP or cultural alignment, this is a useful resource.

The soft spot is exactly where the stress-test note points. The headline "only one model above 50%" is carried by the Persian Legals column, which averages Iran-Law (n=300) and Religion-Rules (n=175). Those two datasets are small, private, and their gold labels are asserted but not externally audited. No inter-annotator agreement, no human-experts ceiling, no item-level error analysis. The authors acknowledge transcription risk in Expert-Eval but not in the legal/religious sets that actually carry the strongest claim. If even a modest share of those labels are ambiguous or keyed to one exam source, the reported 20–52% accuracies could be systematically deflated. So the specific quantitative claim should not be taken at face value.

That said, the broader direction of the paper holds up better. The large Expert-Eval set (49,669 questions) shows a clear gap between closed and open models and between Persian-specific and translated content. The pattern that models do worse on culturally embedded material than on translated general-knowledge tests is consistent across several datasets, not just the two legal ones. So the weakness is in the headline precision, not in the overall finding.

Other concerns are more minor: no confidence intervals anywhere, so small-n differences between models are unquantified; local and API models are scored with different multiple-choice protocols (log-prob vs. structured JSON), which could introduce systematic bias; and the datasets are private, which is defensible for contamination control but makes independent verification impossible. A public validation split or a released sample with gold labels would go a long way.

This paper deserves a serious referee. I would send it to review, but the revision should be asked to release artifacts or at least an audited sample, add a human-performance baseline on Iran-Law and Religion-Rules, and soften the headline claim to match the evidence. The authors are clearly thinking carefully about evaluation methodology, and with those fixes this could be a standard reference for Persian LLM evaluation.

Referee Report

4 major / 6 minor

Summary. The paper introduces MELAC, a Persian-language and Iranian-culture evaluation suite of 19 datasets (13 newly constructed and 6 translated/localized from English benchmarks), and benchmarks 41 LLMs using a standard lm-evaluation-harness-style protocol: log-probability scoring for multiple-choice tasks, regex-based extraction for generative tasks, temperature 0, and private test sets to reduce contamination. The main empirical claim is that LLMs perform poorly on Iranian-specific content: in the Persian Legals category only gpt-4.1-2025-04-14 exceeds 50% accuracy (52.84), which the authors interpret as limited cultural understanding. Secondary findings concern model-family ordering and a positive association between parameter count and performance.

Significance. If the datasets are valid, MELAC is a valuable resource: it broadens culturally grounded evaluation beyond English, covers a wide range of open and API-based models, and uses private test sets to mitigate contamination. The detailed evaluation protocol and the localization effort (rather than mere translation) are clear strengths. However, the headline cultural-gap claim rests on a small, private, and lightly validated subset of the benchmark, so the evidentiary value of the central conclusion is currently conditional rather than established.

major comments (4)

[§5, Table 2, Table 1] The claim that 'all LLMs performed poorly, with only one model among the 41 achieving over 50% accuracy' is not precisely defined. In Table 2, the only category with exactly one model above 50% is Persian Legals (gpt-4.1-2025-04-14 at 52.84), but the surrounding text describes performance on 'Persian-Iranian specific datasets,' and the Persian Linguistic column—also Persian-specific—contains many models above 50% (e.g., 82.25 for gpt-4o). If the claim concerns only the Persian Legals category, it should be stated as such and interpreted with the small sample sizes of Iran-Law (300) and Religion-Rules (175) in mind; if it concerns all Iranian-specific datasets, it is contradicted by Table 2. Please disambiguate the claim and make the basis explicit.
[§3.1, §7, Table 1] The headline conclusion depends on the correctness of gold labels in Iran-Law and Religion-Rules, yet no inter-annotator agreement, item-level audit, or human-expert ceiling is reported for these datasets. The only reliability evidence in the manuscript (Cohen's kappa 0.92 and the translation-quality scores in Appendix D) applies to translated/localized datasets, not to the original legal and religious questions. Section 7 concedes possible transcription errors for Expert-Eval, but no equivalent verification evidence is given for the two small legal/religious sets that carry the central cultural-gap claim. Because the test sets are private, independent checking is impossible. At minimum, the authors should report a human-expert accuracy estimate on a held-out sample of each dataset and a per-item audit of a random subset; otherwise the reported 20–52% accuracies could be systematically deflated by label noise or ambiguity rather than reflecting a true cultural-alignment deficit.
[§4.1, Table 2] No confidence intervals or significance tests are reported for any accuracy or F1 value. Many datasets are small (e.g., Religion-Rules n=175, DC-Homograph n=108), so differences of 1–3 percentage points between models are likely within sampling noise. The secondary claims about model-family ordering and a positive parameter-size correlation should be accompanied by uncertainty estimates, or at least by a statement that differences below a specified threshold are not interpreted. Without such quantification, the model comparisons in Table 2 and the conclusions built on them are stronger than the data support.
[§3, §4.2] The privacy of the test sets is a strength for contamination control, but complete privacy also prevents scientific scrutiny of the gold labels and the scoring pipeline. The paper should release a small public development split with gold labels, plus a data statement for each of the 19 datasets covering provenance, answer-verification procedures, and known ambiguities, while keeping the scored test split private. This would allow the community to audit the construct validity of the benchmark without compromising the contamination protection.

minor comments (6)

[§3.2] The model name 'GPT-o4-mini' appears to be a typo for 'GPT-4o-mini'; please correct it throughout.
[§4.1] The indicator notation 'sk = 1=yk (ˆyk)' is difficult to read; consider using the standard form I(\hat y_k = y_k).
[§3.2] The phrase 'agnatic workflow' appears to be a typo for 'agentic workflow'.
[§3.2] Cohen's kappa of 0.92 is usually described as 'almost perfect' agreement, not 'perfect agreement'; please adjust the wording.
[Table 7] The abbreviation PL in Table 7 is used for Parsi-Lit but is not defined in the table caption, and Table 2 uses the different abbreviation PLing for the same category; please harmonize the notation.
[References] The Hormoz model is cited only as 'Hormoz LLM' with no URL, version, or technical report; a complete reference should be provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MELAC is an empirical benchmark, and the central cultural-gap result is grounded in new expert-authored private datasets; the only self-citation is non-load-bearing.

full rationale

MELAC is an empirical benchmark-construction and evaluation paper, not a derivation; there is no fitted parameter, predicted quantity, or imported uniqueness theorem that could reduce to its own inputs. The headline result—that only one of 41 LLMs exceeds 50% on Persian-Iranian-specific tasks—is computed by applying the standard lm-evaluation-harness scoring to private multiple-choice datasets whose gold labels come from external human sources: Iran-Law was written by three PhD legal scholars with mutual cross-review, Religion-Rules was contributed by clergymen of the relevant faiths, Expert-Eval was transcribed from public Iranian Olympiad, Master's, PhD, and professional license exams, and the translated or localized sets are adapted from public benchmarks such as ARC, MMLU-pro, PIQA, Winogrande, and GSM, with human review and a reported Cohen's kappa of 0.92 for translation quality. No result is defined in terms of another result, and no benchmark score is fed back into dataset construction. The only self-reference is Multiple-Wiki, which reuses the authors' earlier SynTran-Fa general-knowledge dataset; this is a data-source reuse rather than a load-bearing citation, because Multiple-Wiki sits in the General Knowledge category rather than the Persian-Iranian cultural categories and does not carry the cultural-gap conclusion. Section 7's caveat that 'despite a careful verification workflow, the potential for occasional minor errors or inconsistencies remains' is an honest data-quality limitation concerning Expert-Eval; the skeptic's concern that the small, private Iran-Law and Religion-Rules sets lack an inter-annotator agreement or human-ceiling audit is a validity and reproducibility risk, not a circularity, because the low scores are logically consistent with label noise but are not forced by the benchmark's construction. Accordingly, no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on four premises: gold labels are correct, translations preserved meaning, scoring protocols are comparable, and private tests did not leak into training. None of these is independently verifiable from the preprint, and the authors acknowledge transcription-error risk in Limitations. No free parameters or invented entities are needed because the paper reports evaluations rather than derivations.

assumptions (4)

ad hoc to paper Reference answers in all 19 datasets are correct and culturally accurate.
The benchmark scores are only meaningful if gold labels are right. Expert-Eval relies on 30 typists transcribing PDFs, with errors acknowledged in Limitations; ReadingCompQA answers were generated internally; the private test sets prevent independent verification.
domain assumption GPT-o4-mini translation and localization, after human review, preserves meaning and cultural alignment of the six translated datasets.
Section 3.2 states translations were produced by the model and reviewed by three C1 English speakers with Cohen's kappa 0.92, but only average quality scores are reported and no translated samples are released.
domain assumption Log-probability scoring for local models and JSON forced-choice scoring for API models measure the same ability on multiple-choice tasks.
Section 4.1 uses different evaluation protocols for local and API models, so cross-family comparisons in Table 2 assume these protocols are equivalent.
domain assumption The private test sets have not leaked into model training data.
The authors argue privacy avoids contamination, but they cannot verify that API models trained on web data did not see public-source questions, e.g., Expert-Eval from public PDFs or ParsTrivia crawled from websites.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language." pith.science (2026). https://pith.science/paper/KOG76LWS

@misc{pith2026250800673,
  author       = {Pith},
  title        = {Pith review of: MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOG76LWS}},
  note         = {Machine review of arXiv:2508.00673}
}

read the original abstract

As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 6 linked inside Pith

[3]

Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward

Khayyam challenge (persianmmlu): Is your llm truly wise to the persian language? arXiv preprint arXiv:2404.06644 . Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. 2024. Healai: A health- care llm for effective medical documentation. In Proceedings of the 17th ACM International Con- feren...

arXiv 2024
[5]

arXiv preprint arXiv:2408.11857

Hermes 3 technical report. arXiv preprint arXiv:2408.11857. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores- Herr, Joachim K¨ ohler, Ren´ e J¨ akel, and 1 oth- ers. 2024. Towards multilingual llm evalua- tion for european languages. arXiv preprint arXiv:2410.08928. Mar...

arXiv 2024
[6]

arXiv preprint arXiv:2505.09388

Qwen3 technical report. arXiv preprint arXiv:2505.09388. 11 An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guant- ing Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 39 others. 2024a. Qwen2 technical report. ArXiv, abs/24...

arXiv 2024
[2023]

arXiv preprint arXiv:2303.08774

Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, and 1 others. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032 . Andrea Bacciu, Cesare Campagnano...

arXiv 2024
[2024]

arXiv preprint arXiv:2412.04261

Aya expanse: Combining research break- throughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261. EleutherAI. 2021. Multiple choice normalization in LM evaluation. https://blog.eleuther.ai/ multiple-choice-normalization/. Accessed: 2025-07-08. Farhan Farsi, Shahriar Shariati Motlagh, Shayan Bali, Sadra Sabouri, and Saeedeh Momtazi. 2025. P...

arXiv 2021
[2025]

arXiv preprint arXiv:2503.19786

Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mes- nard, Bobak Shahriari, Alexandre Ram´ e, and 1 others. 2024. Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408.00118. Ryan Tekn...

arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [3]

Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward

Khayyam challenge (persianmmlu): Is your llm truly wise to the persian language? arXiv preprint arXiv:2404.06644 . Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. 2024. Healai: A health- care llm for effective medical documentation. In Proceedings of the 17th ACM International Con- feren...

arXiv 2024

[2] [5]

arXiv preprint arXiv:2408.11857

Hermes 3 technical report. arXiv preprint arXiv:2408.11857. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores- Herr, Joachim K¨ ohler, Ren´ e J¨ akel, and 1 oth- ers. 2024. Towards multilingual llm evalua- tion for european languages. arXiv preprint arXiv:2410.08928. Mar...

arXiv 2024

[3] [6]

arXiv preprint arXiv:2505.09388

Qwen3 technical report. arXiv preprint arXiv:2505.09388. 11 An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guant- ing Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 39 others. 2024a. Qwen2 technical report. ArXiv, abs/24...

arXiv 2024

[4] [2023]

arXiv preprint arXiv:2303.08774

Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, and 1 others. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032 . Andrea Bacciu, Cesare Campagnano...

arXiv 2024

[5] [2024]

arXiv preprint arXiv:2412.04261

Aya expanse: Combining research break- throughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261. EleutherAI. 2021. Multiple choice normalization in LM evaluation. https://blog.eleuther.ai/ multiple-choice-normalization/. Accessed: 2025-07-08. Farhan Farsi, Shahriar Shariati Motlagh, Shayan Bali, Sadra Sabouri, and Saeedeh Momtazi. 2025. P...

arXiv 2021

[6] [2025]

arXiv preprint arXiv:2503.19786

Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mes- nard, Bobak Shahriari, Alexandre Ram´ e, and 1 others. 2024. Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408.00118. Ryan Tekn...

arXiv 2024