Pith. sign in

REVIEW 3 major objections 6 minor 38 references

Multilingual Self-Taught Faithfulness Evaluators

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that self-taught synthetic training can produce multilingual faithfulness judges that beat an English-only specialist and approach a model eight times larger.

desk verdict Useful multilingual extension of self-taught evaluators, but the headline results alternate between best-run and mean-run numbers and need a consistent reporting pass before the claims are fully pinned down. read the letter →

arxiv 2507.20752 v1 pith:2U3TPNAO submitted 2025-07-28 cs.CL cs.LG

Carlo Alfano , Aymen Al Marjani , Zeno Jonke , Amin Mantrach , Saab Mansour , Marcello Federico This is my paper
classification cs.CLcs.LG
keywords faithfulnessevaluationLLM-as-judgeself-taughtlearningsyntheticdatamultilingualsummarizationcross-lingualtransferbalancedaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multilingual faithfulness evaluators can be trained without human labels, using synthetic summaries and a self-taught loop, and that such a model can outperform an English-only state-of-the-art evaluator on average across languages while coming close to a model eight times its size. The best result is a 9-billion-parameter model that reaches 78.5 average balanced accuracy on multilingual benchmarks, compared with 76.1 for Bespoke-MiniCheck-7B and 80.0 for Qwen2.5-72B-Instruct. Training on English data alone transfers well to other languages, provided the starting model is proficient in those languages. A sympathetic reader would care because it suggests a scalable, label-free path to judging factual consistency in many languages.

What carries the argument

The load-bearing mechanism is the self-taught filtering loop: an auxiliary LLM produces faithful summaries and corrupt summaries (the indirect corruption strategy corrupts the document first, then summarizes it), the current judge labels each sentence, and only judgments that agree with the pseudo-labels enter the fine-tuning set; this repeats across iterations, so the judge learns from its own accepted predictions. The indirect corruption strategy matters because it lets a relatively small model generate reliable unfaithful sentences by summarizing a deliberately corrupted document, rather than having to insert specific errors into a faithful summary.

What would settle it

Compare the best trained evaluator against human labels on a multilingual set whose corrupt summaries are produced by a different method than the one used in training; if accuracy drops sharply, the model is matching training artifacts rather than judging faithfulness. A direct test is whether the evaluator flags summaries that state something true but outside the document - the paper's own 'out-of-context' category - on documents whose supporting passages contain that information.

Watch

Extended reading notes

Core claim

The central claim is that the STEMF pipeline, which generates faithful and corrupted summaries with an auxiliary LLM, asks the current evaluator to judge them, keeps only judgments that match the generated pseudo-labels, and fine-tunes on those accepted judgments, produces a multilingual faithfulness judge that outperforms the English-specialized Bespoke-MiniCheck-7B and approaches Qwen2.5-72B-Instruct. The paper reports 78.5 average balanced accuracy for its best gemma-2-9b-it based evaluator, and 87.7 sentence-wise balanced accuracy on English FRANK, above the 86.4 reported for GPT-4 in prior work. The paper also establishes that the gain from training is tied to the starting model's general language ability, and that translating to English before judging often hurts rather than helps.

Load-bearing premise

The whole pipeline assumes that the synthetic 'faithful' and 'corrupt' summaries generated by the auxiliary LLM really are faithful and unfaithful, and that the self-filtering step does not amplify systematic mistakes in those pseudo-labels.

Editorial extensions

If this is right

  • A 9B-parameter evaluator trained without human labels can replace much larger prompt-based judges for multilingual faithfulness checking in practice.
  • English-only training data is sufficient to improve evaluation across German, French, Spanish, Hindi, Arabic, and Italian, so the method does not require per-language training corpora.
  • The link between starting-model language proficiency and the size of the improvement gives a practical selection rule: pick a base model strong in the target language, then train on English.
  • Training only the central layers yields comparable accuracy after five iterations while roughly halving the memory needed for LoRA adapters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the correlation between improvement and starting-model proficiency is causal, the framework's ceiling is set by the base model's multilingual ability; a stronger base model should push the average closer to or beyond the 72B baseline.
  • The success of English-only training hints that faithfulness features are largely language-agnostic in these models; a direct testable extension is probing whether the learned evaluator transfers to low-resource languages not in the benchmark set.
  • Because the filter keeps only judgments the judge already gets right, the loop can silently entrench systematic errors in the pseudo-labels; an external check on a diverse corruption distribution would show how much of the gain is genuine faithfulness detection versus artifact-matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents STEMF (Self-Taught Evaluators for Multilingual Faithfulness), a framework for training LLM-based faithfulness evaluators using synthetic multilingual summarization data, with knowledge of the faithful/corrupt labels supplied by an auxiliary LLM. The evaluator is fine-tuned iteratively, accepting only its own judgments that agree with the synthetic pseudo-labels. The authors claim that a gemma-2-9b-it model trained with this approach on English-only data outperforms Bespoke-MiniCheck-7B on average across a multilingual benchmark suite and approaches Qwen2.5-72B-Instruct, and that it reaches 87.7 sentence-wise balanced accuracy on English FRANK, exceeding the 86.4 reported for GPT-4 by Song et al. (2024). The paper also studies the effect of training-language choice, corruption strategy (direct vs. indirect), central-layer-only fine-tuning, proxy NLI data, and translation-based inference.

Significance. If the claims hold, the result is practically significant: it suggests that a 9B-parameter model can be turned into a competitive multilingual faithfulness judge without any human-annotated training data, which would substantially lower the cost of building such evaluators. The paper's methodology is clearly described, includes three training runs with standard errors (Table 9), and makes its prompts and hyperparameters available in the appendices, which is a strength. The empirical finding that training on English is usually the best or near-best choice across test languages, and the correlation with the starting model's MMMLU proficiency, are actionable insights for practitioners. The main weakness is in the reporting and statistical validation of the headline comparisons, which currently mix best-run and average-run numbers without reconciliation.

major comments (3)
  1. The central claim that both STEMF-trained models outperform Bespoke-MiniCheck-7B on average is not supported by a consistent choice of evaluation number. Section 5 states that both trained models outperform MiniCheck on average, which is true for Table 1's best-run numbers (78.5 and 77.4 vs. 76.1), but Table 9's three-run means are 77.5±0.8 for gemma-2-9b-it+STEMF and 75.7±1.3 for Qwen2.5-7B+STEMF, the latter being below MiniCheck's 76.1. Similarly, the prose reports a sentence-wise balanced accuracy of 87.7 for gemma on English FRANK, but Table 1 lists 86.8 and Table 9 gives 85.7±0.7. These discrepancies are load-bearing for the paper's headline. Please report all comparisons using mean±standard error consistently, and provide a statistical test (e.g., paired or unpaired with proper accounting for run-level variability) for the claim that STEMF models outperform MiniCheck.
  2. The non-English evaluation partly relies on machine-translated FRANK for German, Spanish, French, Arabic, and Italian, while Hindi is excluded from translated FRANK due to low translation quality. However, the manuscript does not report results separately for native benchmarks (MEMERAG, mFACE) versus translated FRANK, nor does it analyze the sensitivity of the reported averages to translation quality. Since the claim is about multilingual performance, the average balanced accuracy in Table 1 and Table 9 mixes native and translated data. Please report per-benchmark results and show that the headline conclusions are not driven by noise from the translated portion. In addition, Hindi is evaluated only on MEMERAG, leaving it with a single benchmark; the paper should either add the native mFACE Hindi evaluation or acknowledge this limitation explicitly in the main text.
  3. The human-label control for quality assurance is run only for the 'all' language mixture, not for the English-only setting that produces the headline results. The manuscript correctly notes that it cannot guarantee the correctness of the faithful/corrupt summaries generated by M, and Step 2 of the framework keeps only judgments that agree with the pseudo-labels, so systematic pseudo-label errors could be amplified over iterations. The existing control is informative but not sufficient to validate the English-only training condition. Please run the human-label control (or an equivalent pseudo-label accuracy check on a held-out English sample) for the English-only condition, or explicitly discuss why the result is expected to transfer from the 'all' condition.
minor comments (6)
  1. The abstract says the framework shows 'improvements over existing baselines, including state-of-the-art English evaluators,' but the mean results in Table 9 show Qwen2.5-7B+STEMF at 75.7 is below MiniCheck's 76.1. Please qualify this statement or adjust the abstract to reflect the run-level variability.
  2. The sentence-wise balanced accuracy comparison to GPT-4 (87.7 vs. 86.4) is taken from Song et al. (2024) without verifying that the evaluation protocol is identical to the one used here. Please either reproduce GPT-4 under the same protocol or drop the direct comparison, especially given that 87.7 is itself not the value reported in Tables 1 and 9.
  3. The indexing of the starting model is inconsistent: Algorithm 1 initializes J0 ← M, while the text denotes J1 as the starting LLM. Please harmonize the notation (e.g., by defining J0 as the base model and Ji as the model after i iterations).
  4. The standard errors in Table 9 are reported per column, but the Avg column combines results across benchmarks with different sizes (see Table 10). Please report the number of evaluation examples per benchmark and, if possible, compute the average using a consistent weighting scheme to make the standard error interpretable.
  5. The translation-based baselines are evaluated only for starting models, not for the STEMF-trained models. The abstract's wording that the framework improves over 'machine translation-based approaches' is therefore not directly supported by the experiments, which only show that translation tends to hurt the base models. Please clarify this distinction in the text.
  6. The caption says 'balanced accuracy (y-axis) is averaged over all benchmarks and languages,' but the y-axis label reads 'BAcc Increase.' The y-axis actually shows the improvement over the starting model; please correct the caption or the axis label to avoid confusion.

Circularity Check

0 steps flagged · score 1.0 of 10

No construction-level circularity: STEMF is evaluated on held-out human benchmarks, and the self-training filter does not make the reported predictions a restatement of the training inputs.

full rationale

STEMF's training loop is self-referential by design: in Step 2, judgments from the current evaluator J_i are accepted only when the predicted label matches the pseudo-label produced in Step 1, and Step 3 fine-tunes on those accepted judgments. This is a bootstrapping procedure, not a circular derivation: the training labels are LLM-generated pseudo-labels, whereas the paper's central claims are tested on separate, human-annotated benchmarks (MEMERAG, mFACE, FRANK). No equation in the paper defines the reported balanced accuracy in terms of the training pseudo-labels, and the held-out evaluation is independent of the filter. The paper itself flags the key limitation: 'we cannot guarantee the correctness of faithful summaries or the unfaithfulness of corrupt summaries when they are generated by M in the base methodology' (Section 3.2), and the human-labels variation is a control for exactly this concern. The only author-overlap citation used as an external reference is FineSurE (Song et al., 2024) for the GPT-4 FRANK number (86.4) and the prompt implementation; that citation is not load-bearing for the main result, which is established by comparing trained models against MiniCheck and Qwen2.5-72B on externally available benchmarks. The discrepancy between Table 1 (best-run numbers, e.g., 78.5 and 87.7 in prose) and Table 9 (mean-run numbers, e.g., 77.5±0.8 and 85.7±0.7) is a reporting/reproducibility issue, not evidence that the prediction reduces to the input. No self-definitional, fitted-input-called-prediction, uniqueness-imported, or ansatz-smuggled steps were found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework relies on several domain assumptions about the reliability of synthetic data, evaluation benchmarks, and self-training dynamics. No new physical or conceptual entities are introduced. Training hyperparameters are standard choices, not fitted to the test sets.

free parameters (5)
  • Number of training iterations r = 5 (also tested 1 and 3)
    Hand-chosen stopping point; Table 8 shows 3 and 5 are similar, 1 is worse.
  • Documents sampled per iteration = 1000
    Step 0 randomly samples one thousand documents from Omega.
  • LoRA rank / alpha = 128 / 256
    Standard hyperparameters from Table 5, not fitted to evaluation benchmarks.
  • Learning rate = 5e-5
    Standard hyperparameter from Table 5.
  • Batch size / gradient accumulation = 4 / 16
    Standard hyperparameters from Table 5.
assumptions (5)
  • domain assumption Pseudo-labels from the auxiliary LLM (faithful vs. corrupt summaries) are accurate enough to train a useful evaluator.
    Section 3.2 acknowledges no guarantee of correctness; the human-label control partially validates this only for one language mix.
  • domain assumption Machine-translated FRANK is a valid evaluation benchmark for target languages.
    Section 4 uses translated FRANK for Arabic, German, Spanish, French, and Italian, and excludes Hindi due to low translation quality. Translationese may bias the measured transfer.
  • domain assumption Balanced accuracy on MEMERAG, mFACE, and FRANK correctly measures faithfulness evaluation quality.
    The paper uses only balanced accuracy and relies on the benchmarks' human labels without independent verification.
  • domain assumption MMMLU scores are a valid proxy for language proficiency relevant to faithfulness evaluation.
    Figure 3 uses MMMLU to explain improvements; this assumes the proxy transfers to the evaluation task.
  • domain assumption Self-training with filtered judgments does not collapse or overfit to self-agreement.
    Step 2 selects only judgments that agree with pseudo-labels; the paper does not analyze distribution shift across iterations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Self-Taught Faithfulness Evaluators." pith.science (2026). https://pith.science/paper/2U3TPNAO

@misc{pith2026250720752,
  author       = {Pith},
  title        = {Pith review of: Multilingual Self-Taught Faithfulness Evaluators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2U3TPNAO}},
  note         = {Machine review of arXiv:2507.20752}
}
read the original abstract

The growing use of large language models (LLMs) has increased the need for automatic evaluation systems, particularly to address the challenge of information hallucination. Although existing faithfulness evaluation approaches have shown promise, they are predominantly English-focused and often require expensive human-labeled training data for fine-tuning specialized models. As LLMs see increased adoption in multilingual contexts, there is a need for accurate faithfulness evaluators that can operate across languages without extensive labeled data. This paper presents Self-Taught Evaluators for Multilingual Faithfulness, a framework that learns exclusively from synthetic multilingual summarization data while leveraging cross-lingual transfer learning. Through experiments comparing language-specific and mixed-language fine-tuning approaches, we demonstrate a consistent relationship between an LLM's general language capabilities and its performance in language-specific evaluation tasks. Our framework shows improvements over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches.

Figures

Figures reproduced from arXiv: 2507.20752 by the authors.

Figure 1
Figure 1. Our framework for self-taught multilingual evaluators [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A comparison of training languages combinations across various LLMs. The plot reports the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Relationship between performance improvement and performance of the starting LLM across the different [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scheme for producing corrupted sentences, given a document and faithful sentences. Replaces the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 12 canonical work pages

  1. [1]

    Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023. https://doi.org/10.18653/v1/2023.findings-acl.220 Multilingual summarization with factual consistency evaluation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3562--3591, Toronto, Canada. Association for Computational ...

    work page doi:10.18653/v1/2023.findings-acl.220 2023

  2. [2]

    Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.399 Revisiting machine translation for cross-lingual classification . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6489--6499, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.emnlp-main.399 2023

  3. [3]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  4. [4]

    Andrea Bacciu, Cesare Campagnano, Giovanni Trappolini, and Fabrizio Silvestri. 2024. https://aclanthology.org/2024.lrec-main.388/ D ante LLM : Let`s push I talian LLM research forward! In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4343--4355, Torino, Ital...

    work page 2024

  5. [5]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. CoRR

    work page 2024

  6. [6]

    Kathrin Blagec, Georg Dorffner, Milad Moradi, Simon Ott, and Matthias Samwald. 2022. https://doi.org/10.18653/v1/2022.nlppower-1.6 A global analysis of metrics used for measuring performance in natural language processing . In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 52--63, Dublin, Ireland. Association for Comp...

    doi:10.18653/v1/2022.nlppower-1.6 2022

  7. [7]

    Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. 2023. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems, 36:31543--31557

    work page 2023

  8. [8]

    Bowman, Holger Schwenk, and Veselin Stoyanov

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    work page 2018

Show all 38 references
  1. [9]

    María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, and Marcello Federico. 2025. https://arxiv.org/abs/2502.17163 Memerag: A multilingual end-to-end meta-evaluation benchmark for retrieval augmented generation . Preprint, arXiv:2502.17163

    2025 arXiv

  2. [10]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL -sum: Large-scale multilingual abstractive summarization for 44 languages . In Fin...

    2021 doi

  3. [11]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28

    2015

  4. [12]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  5. [13]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. https://openreview.net/forum?id=8euJaTveKw Prometheus: Inducing fine-grained evaluation capability in language models . In...

    2024

  6. [14]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  7. [15]

    Bespoke Labs. 2024. https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B Bespoke-minicheck-7b

    2024

  8. [16]

    Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.360 W iki L ingua: A new benchmark dataset for cross-lingual abstractive summarization . In Findings of the Association for Computational Linguistics: EMNLP 2...

    2020 doi

  9. [17]

    Surafel M Lakew, Quintino F Lotito, Matteo Negri, Marco Turchi, and Marcello Federico. 2017. Improving zero-shot translation of low-resource languages. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 113--119

    2017

  10. [18]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  11. [19]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  12. [20]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don`t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natura...

    2018 doi

  13. [21]

    Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. https://doi.org/10.18653/v1/2021.naacl-main.383 Understanding factuality in abstractive summarization with FRANK : A benchmark for factuality metrics . In Proceedings of the 2021 Conference of the North America...

    2021 doi

  14. [22]

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. https://doi.org/10.1162/tacl_a_00660 Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies . Transactions of the Associ...

    2024 doi

  15. [23]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  16. [24]

    Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. https://doi.org/10.18653/v1/2024.naacl-long.462 Branch-solve-merge improves large language model evaluation and generation . In Proceedings of the 2024 Conference of the North American...

    2024 doi

  17. [25]

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. https://doi.org/10.18653/v1/2024.acl-long.51 F ine S ur E : Fine-grained summarization evaluation using LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics...

    2024 doi

  18. [26]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.499 M ini C heck: Efficient fact-checking of LLM s on grounding documents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818-...

    2024 doi

  19. [27]

    Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu ' an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024 b . https://doi.org/10.18653/v1/2024.naacl-long.251 T ofu E val: Evaluating hallucinatio...

    2024 doi

  20. [28]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.309 Language-specific neurons: The key to multilingual capabilities in large language models . In Proceedings of the 6...

    2024 doi

  21. [29]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. https://github.com/huggingface/alignment-handbook The Alignment Handbook

  22. [30]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.620 Interpretable preferences via multi-objective reward modeling and mixture-of-experts . In Findings of the Association for Computational Linguistics: E...

    2024 doi

  23. [31]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024 b . Self-taught evaluators. arXiv preprint arXiv:2408.02666

    2024 arXiv

  24. [32]

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024 c . https://openreview.net/forum?id=5Nn2BLV7SB Panda LM : An automatic evaluation benchmark for LLM...

    2024

  25. [33]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://doi.org/10.18653/v1/2024.acl-long.820 Do llamas work in E nglish? on the latent language of multilingual transformers . In Proceedings of the 62nd Annual Meeting of the Association for Computati...

    2024 doi

  26. [34]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594

    2024 arXiv

  27. [35]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning

    2024

  28. [36]

    Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 a . Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055

    2024 arXiv

  29. [37]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024 b . How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815

    2024 arXiv

  30. [38]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.