Pith. sign in

REVIEW 5 major objections 6 minor 22 references

Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a Persian-specialized sentence-embedding model, created by continual pretraining and contrastive fine-tuning, retrieves relevant context for Persian question-answering better than generic multilingual embeddings…

desk verdict A substantial Persian RAG resource paper whose central superiority claim is undermined by an internally inconsistent Table 5; worth refereeing but not citable as-is. read the letter →

arxiv 2501.04858 v1 pith:OAFZXP2J submitted 2025-01-08 cs.CL

Sara Bourbour Hosseinbeigi , Sina Asghari , Mohammad Ali Seif Kashani , Mohammad Hossein Shalchian , Mohammad Amin Abbasi This is my paper · ORCID
classification cs.CL
keywords Retrieval-AugmentedGenerationPersianNLPSentenceEmbeddingsLow-ResourceLanguagesBenchmarkingLanguageModelFine-TuningQuestionAnsweringDocumentSummaryIndexing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation (RAG) for Persian is underdeveloped, and this paper sets out to close that gap with a Persian-specific embedding model and a benchmark. The authors build a masked language model on a 73.11-billion-token Persian corpus, fine-tune it into a sentence-embedding model, and evaluate both on three Persian datasets that cover general knowledge, scientific text, and formal policy documents. Their central claim is that the resulting embeddings retrieve more relevant context than multilingual embedding models, and that generation accuracy improves when retrieval is paired with tuned settings such as low temperature, smaller chunks, and document-summary indexing. If the claim holds, it gives practitioners a concrete recipe for building RAG systems in Persian and a template for other low-resource languages.

What carries the argument

The load-bearing object is MatinaSRoberta, a Persian sentence-embedding model made in two stages. First, a multilingual masked language model is continually pretrained on 73.11 billion Persian tokens gathered from scientific articles, books, social media, websites, and a large web-crawl collection, creating the base encoder MatinaRoberta. Second, that encoder is fine-tuned with sentence-level objectives—Multiple Negatives Ranking Loss, Contrastive Loss, Softmax Loss, and Triplet Loss—over a mixed Persian and multilingual set of paired and triplet examples. The resulting dense 1024-dimensional embeddings are the mechanism that is supposed to capture Persian-specific morphology, flexible word order, and agglutination, which is what the paper argues gives it the retrieval advantage.

What would settle it

Inspect the largest reported first-retrieval number for the general-knowledge set: if 'accuracy' means the percentage of queries, a value such as 8,231 cannot be correct for a test set described as 8,002 questions, so recomputing the table with the paper's stated weights and explicit denominators settles the claimed ranking.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that training a sentence-embedding model on a large, native Persian corpus—rather than relying on multilingual embeddings—yields consistently better retrieval for Persian retrieval-augmented-generation (RAG) pipelines. The fine-tuned model uses 1024-dimensional embeddings produced by pooling token representations, trained with multiple ranking and contrastive losses on question-answer, entailment, paraphrase, and triplet data. The authors report that on the general-knowledge and scientific-specialized datasets this model ranks the correct context above multilingual baselines, and that on the formal organizational dataset it remains competitive. They also report that larger generator models produce the most accurate answers, and that retrieval-side choices—temperature, chunk size, and summary indexing—materially affect generation quality.

Load-bearing premise

The central claim assumes that the retrieval scores in Table 5 are computed as defined in Section 5.5.1, yet the reported numbers look like raw counts rather than percentages, so without knowing the normalization the model ranking is not interpretable.

Editorial extensions

If this is right

  • Persian search engines and document-analysis tools can adopt these embeddings and expect more reliable retrieval than with multilingual defaults.
  • The two-stage recipe of continued pretraining on a native corpus followed by multi-loss sentence fine-tuning becomes a reusable template for other low-resource languages.
  • Practical RAG settings can be set with more confidence: a temperature near 0.25, a chunk size near 512 tokens for formal or technical text, and document summary indexing for long documents.
  • The three-dataset benchmark gives future Persian RAG models a standard frame for reporting retrieval and generation quality.
  • If the gains hold, smaller models fine-tuned this way could reduce the need to rely on very large generator models for low-resource RAG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Ablating the two training stages would isolate how much of the retrieval gain comes from the Persian pretraining step versus the sentence fine-tuning step, which the paper does not test directly.
  • Comparing MatinaSRoberta against a multilingual sentence-embedding model fine-tuned with the same losses on the same pair data would show whether the advantage is language-specific or just an effect of extra fine-tuning.
  • Because the organizational-report evaluation uses a single policy document, the apparent formal-domain advantage may not generalize to other legal or bureaucratic Persian text until tested on a broader sample.
  • Applying the same data-and-fine-tuning recipe to a second low-resource language would be the most direct test of the paper's claim that its approach transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper develops Persian-specific language models for retrieval-augmented generation: MatinaRoberta (continual pretraining of XLM-RoBERTa Large on 73.11B Persian tokens) and MatinaSRoberta (Sentence-BERT-style fine-tuning of MatinaRoberta with multiple loss functions). It introduces three Persian RAG benchmarks (PQuad, a GPT-4-generated scientific-specialized dataset, and a GPT-4o-generated organizational-report dataset) and evaluates embedding models and LLMs under various RAG configurations. The headline claim is that the Persian-specialized embedding MatinaSRoberta consistently outperforms multilingual baselines across all three datasets in retrieval accuracy (Table 5), and that larger LLMs such as Llama-3.1-70B achieve the best generation quality (Tables 6, 10-12). The paper also reports ablation-style experiments on temperature tuning, chunk size, and document summary indexing.

Significance. If the central claims were fully established, the paper would make a useful practical contribution to Persian NLP and low-resource RAG: a new pretrained masked language model and sentence embedding model, benchmarks in three domains, and concrete configuration guidance (chunk size, temperature, summary indexing). The authors also provide a sizeable pretraining corpus description. However, the significance is currently capped by three unresolved issues: the central retrieval comparison in Table 5 is internally inconsistent with the stated metric definitions; the fine-tuning corpus overlaps with the evaluation data without any stated held-out split; and the scientific and organizational datasets are generated and evaluated by LLM-based pipelines without manual verification. Additionally, the paper does not release code, models, or datasets, which limits reproducibility of the quantitative claims. These issues must be resolved before the contributions can be assessed reliably.

major comments (5)
  1. [Table 5 / Section 5.5.1] The central superiority claim for MatinaSRoberta relies on Table 5, but that table cannot be interpreted under the metric definitions in Section 5.5.1. First Result Accuracy is defined as 'the percentage of queries where the correct answer was retrieved as the first result,' yet every PQUAD entry exceeds 100%: MatinaSRoberta reports 8,231, Ahd 7,231, LaBSE 4,713, L12-V2 4,087, Qwen2-7 3,979, Alibaba/gte-large 1,341, Alibaba/gte 814, while Section 5.1.1 fixes the PQuad test set at 8,002 questions. If these figures are raw correct-retrieval counts, they contradict the per-query 'first result' definition and still cannot exceed the number of test queries; if they are percentages, they are out of range. The analogous issue appears in the Scientific-Specialized and Organizational Report columns, where denominators are not specified anywhere. The weighted Avg formula (3*first + 2*second + third) also does not reproduce any reported Average: for example, MatinaSRoberta on PQuad would give (3*8231 + 2*475 + 142)/? = 47.70 only if divided by a denominator near 540, which is not stated. The paper must either correct the table to report interpretable percentages with explicit denominators, or provide the retrieval code and raw outputs so the reader can reproduce the ranking. As written, the headline comparison is unverifiable.
  2. [Table 5 / Section 5.2] Table 5 introduces a baseline named 'Ahd' that is never defined in Section 5.2 (which lists only MatinaSRoberta, LaBSE, L12-V2, Qwen2-7, Alibaba/gte-large, and Alibaba/gte). This is a load-bearing omission because Ahd outperforms MatinaSRoberta on the Organizational Report dataset on the Avg metric (22.52 vs. 21.02), directly contradicting the Section 6.1 statement that 'MatinaSRoberta model consistently demonstrated superior performance across all three datasets.' The paper must either identify what 'Ahd' is (architecture, checkpoint, training data) and include it in Section 5.2, or remove it from Table 5 and re-evaluate the consistency claim quantitatively. As presented, the table and the textual claim are mutually inconsistent.
  3. [Sections 4 and 5.1.1 / Table 5] There is a circularity risk that is not addressed: Section 4 states that fine-tuning MatinaSRoberta used 'QA pairs from the PQuad collection,' while Section 5.1.1 and Table 5 use PQuad as the general-knowledge evaluation dataset. The paper never states whether the PQuad QA pairs used for fine-tuning were held out from the 8,002-question test set used in Table 5. If the same PQuad items (or their source Wikipedia passages) appear in both training and testing, the reported PQuad retrieval results are partly in-domain and the superiority claim is inflated. The authors must specify the exact split, the overlap-removal procedure between fine-tuning data and evaluation test, and re-run Table 5 on a genuinely held-out set if needed.
  4. [Sections 5.1.2, 5.1.3, 5.4, Tables 10-12] The scientific-specialized and organizational-report datasets are generated by GPT-4 and GPT-4o, respectively, and the RAGAS evaluation metrics are computed by LLM-based judges (faithfulness, answer relevancy, context precision, context recall). The paper provides no manual validation of the MCQ ground-truth answers, no human agreement study on the RAGAS judgments, and no reporting of the judge LLM used for RAGAS. Given that the same types of models are being evaluated, the possibility of systematic judge bias or dataset artifacts cannot be excluded. The authors should release the generated datasets, report a human-validated subset agreement rate for both MCQ correctness and RAGAS scores, and state which LLM served as the RAGAS judge. Without this, the generation-quality conclusions in Tables 10-12 and the optimization findings in Tables 7-9 rest on unverified auto-generated ground truth.
  5. [Tables 10-12] The RAGAS tables show that Context Precision is constant within each dataset (0.7750 for PQUAD, 0.8138 for Scientific-Specialized, and 0.5355 for two models in Organization Report), which is suspicious for a metric that is supposed to measure how well the retriever ranks relevant items for each query. The paper does not explain why the ranking quality is identical across all LLMs and embedding configurations. If Context Precision is computed from the same retrieved set for all models, then it is not a model-level metric as implied by the prose, or the reported values are rounded to four significant digits and coincidentally equal, which still requires explanation. The authors should clarify the computation pipeline and report the non-rounded values and any variance across evaluation runs.
minor comments (6)
  1. [Section 3.1 / Table 1] Table 1 lists token counts with a stated sum of 73.11B, but the rows sum to 73.11B only if 49.6 + 14.78 + 2.34 + 2.84 + 3.55 = 73.11; the values do sum correctly, yet the paper earlier in Section 3 says '54.69 billion Persian tokens,' which is inconsistent with the Table 1 sum of 73.11 billion. Please reconcile the corpus size.
  2. [Table 3 title] Table 3 is captioned 'Token Distribution Across Pretraining Datasets' but its rows describe fine-tuning hyperparameters for MatinaSRoberta (batch size, epochs, warmup ratio, etc.). The caption appears to be a copy-paste error and should be corrected.
  3. [Equation / Section 5.5.1] The Overall Score formula is described in words as weightings (3, 2, 1) for first/second/third result accuracy, but no explicit formula is given. Since Table 5's Avg column cannot be reproduced, the paper should provide the exact formula, including the denominator and how unanswerable questions are handled.
  4. [Tables 7-9] Tables 7-9 report accuracy values without error bars or statistical tests, and the differences are often small (e.g., temperature 0 vs. 0.25 on PQUAD: 73.25 vs. 72.83). Given the sample sizes (8,002 PQuad questions), the paper should report confidence intervals or at least a paired significance test to support claims such as '0.25 provided the optimal balance' and 'smaller chunk size of 512 tokens produced the best overall results.'
  5. [Section 5.2] The embedding model list includes 'Alibaba/gte' and 'Alibaba/gte-large,' but section 6.1 later refers to 'OpenAI embeddings' that are not listed in Section 5.2 or in Table 5. Please align the model descriptions with the actual evaluated models and remove the stray reference.
  6. [Section 6.1] The prose says 'MatinaSRoberta onsistently demonstrated the highest performance across most datasets' (typo 'onsistently'), and Section 6.1 claims superiority on the scientific-specialized dataset while Table 5 shows L12-V2 having a higher Avg (29.23) than MatinaSRoberta (42.23? no, 42.23 > 29.23, so this is fine only if the reader checks the numbers; the inconsistency with Ahd on Organizational Report remains as stated in the major comment). Please proofread and, more importantly, re-evaluate the 'consistent superiority' claim in light of the corrected Table 5.

Circularity Check

1 steps flagged · score 4.0 of 10

PQuad fine-tuning/evaluation overlap makes one of the three legs of the superiority claim partly in-domain; the other two datasets provide independent content, so the claim is only partially circular.

  1. fitted input called prediction [Section 4 (fine-tuning) and Sections 5.1.1/6.1, Table 5]
    "Domain-specific Persian datasets played a critical role in enriching the model’s comprehension of specialized settings. These included QA pairs from the PQuad collection... MatinaSRoberta model consistently demonstrated superior performance across all three datasets, particularly in the PQuad and scientific-specialized datasets."

    MatinaSRoberta was fine-tuned on QA pairs from the PQuad collection, and PQuad is also one of the three evaluation datasets. The paper never states that fine-tuning was restricted to the 63,994-example training split, so the PQuad column of Table 5 may measure retrieval of content the model was explicitly fitted to. The claimed 'superior... retrieval accuracy across datasets' therefore has one leg that is, at least partly, an in-domain fit rather than an independent prediction. The scientific-specialized and organizational-report datasets are not listed among the fine-tuning sources, so the claim retains independent content there.

full rationale

The central load-bearing claim in Section 6.1 is that MatinaSRoberta outperforms baselines 'across all three datasets.' The PQuad leg of this claim is not independent: Section 4 lists 'QA pairs from the PQuad collection' among the fine-tuning data, and Section 5.1.1 defines PQuad as an evaluation dataset with training/validation/test splits, but no statement says the fine-tuning used only the training split. In the absence of that stated split, the PQuad results in Table 5 are partly a measure of fitting rather than prediction. This is a genuine fitted-input-called-prediction pattern for one of the three datasets. The other two datasets are not part of the fine-tuning corpus, so the 'across all three datasets' claim still has independent content. There are also serious non-circularity reporting defects: First Result Accuracy is defined as a percentage, yet Table 5 reports 8,231 for PQuad against an 8,002-question test set; the stated weighted Overall Score formula does not reproduce any reported Avg; and Ahd and LaBSE have identical raw counts but different Avg in the Organizational Report column. These prevent the table from being read as an external benchmark, but they are correctness issues rather than circularity. The self-citations to PersianLLama and AriaBERT are not load-bearing for the paper's central derivation, so they do not raise the score. Overall, the superiority claim is partially circular because one of its three supporting legs reduces to the model's own training data, while the remaining legs and the model-building pipeline are self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on evaluation metrics and dataset assumptions rather than a formal derivation. The most substantive burden is the unstated handling of PQuad in both training and evaluation, the reliance on unvalidated LLM-generated datasets and LLM-based judges, and arbitrary weights in the composite retrieval score. No new theoretical entities are introduced.

free parameters (3)
  • Retrieval score weights for Overall/Avg metric = 3, 2, 1
    Introduced in Section 5.5.1 without justification; the composite score used to rank embedding models depends on these arbitrary weights, and no denominator for the Avg column is given.
  • Fine-tuning hyperparameters for MatinaSRoberta = batch 30, epochs 4, warmup 0.4, polynomial LR schedule (Table 3)
    Chosen by hand for training; they are not tied to any selection procedure and could affect retrieval quality, though they are not the main concern.
  • RAG configuration (temperature, chunk size, top-k) = temperature 0.25, chunk 1024, overlap 256, top-k 5
    Baseline evaluation settings in Table 4; the best-practice claims are derived from hand-selected candidates rather than a systematic, fully described search.
assumptions (4)
  • domain assumption RAGAS metrics computed by an LLM judge are valid proxies for human retrieval and generation quality.
    Invoked in Sections 5.4 and 5.5.2 with no human correlation study or validation on Persian.
  • domain assumption GPT-4 and GPT-4o generated correct multiple-choice questions and answers for the scientific and organizational datasets.
    Used in Sections 5.1.2 and 5.1.3 with no manual verification or inter-annotator agreement reported.
  • domain assumption PQuad used in fine-tuning (Section 4) is kept disjoint from PQuad used in evaluation (Section 5.1.1).
    The paper does not state that the fine-tuning split excludes the evaluation split; if not disjoint, the PQuad ranking is circular.
  • domain assumption XLM-RoBERTa Large is a suitable initialization and continuing pretraining on the described corpus improves Persian ability.
    Improvements are attributed to the Persian pretraining corpus in Section 3, but no ablations over base model or corpus are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization." pith.science (2026). https://pith.science/paper/OAFZXP2J

@misc{pith2026250104858,
  author       = {Pith},
  title        = {Pith review of: Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAFZXP2J}},
  note         = {Machine review of arXiv:2501.04858}
}
read the original abstract

This paper examines the specific obstacles of constructing Retrieval-Augmented Generation(RAG) systems in low-resource languages, with a focus on Persian's complicated morphology and versatile syntax. The research aims to improve retrieval and generation accuracy by introducing Persian-specific models, namely MatinaRoberta(a masked language model) and MatinaSRoberta(a fine-tuned Sentence-BERT), along with a comprehensive benchmarking framework. Three datasets-general knowledge(PQuad), scientifically specialized texts, and organizational reports, were used to assess these models after they were trained on a varied corpus of 73.11 billion Persian tokens. The methodology involved extensive pretraining, fine-tuning with tailored loss functions, and systematic evaluations using both traditional metrics and the Retrieval-Augmented Generation Assessment framework. The results show that MatinaSRoberta outperformed previous embeddings, achieving superior contextual relevance and retrieval accuracy across datasets. Temperature tweaking, chunk size modifications, and document summary indexing were explored to enhance RAG setups. Larger models like Llama-3.1 (70B) consistently demonstrated the highest generation accuracy, while smaller models faced challenges with domain-specific and formal contexts. The findings underscore the potential for developing RAG systems in Persian through customized embeddings and retrieval-generation settings and highlight the enhancement of NLP applications such as search engines and legal document analysis in low-resource languages.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

  1. [1]

    Natural language processing,

    K. Chowdhary and K. Chowdhary, "Natural language processing," Fundamentals of artificial intelligence, pp. 603-649, 2020

    2020

  2. [2]

    Retrieval-augmented generation for large language models: A survey,

    Y. Gao et al., "Retrieval-augmented generation for large language models: A survey," arXiv preprint arXiv:2312.10997, 2023

    arXiv 2023

  3. [3]

    Large language models: A survey,

    S. Minaee et al., "Large language models: A survey," arXiv preprint arXiv:2402.06196, 2024

    arXiv 2024

  4. [4]

    A comprehensive survey of hallucination mitigation techniques in large language models,

    S. Tonmoy et al., "A comprehensive survey of hallucination mitigation techniques in large language models," arXiv preprint arXiv:2401.01313, 2024

    arXiv 2024

  5. [5]

    Persianllama: Towards building first persian large language model,

    M. A. Abbasi, A. Ghafouri, M. Firouzmandi, H. Naderi, and B. M. Bidgoli, "Persianllama: Towards building first persian large language model," arXiv preprint arXiv:2312.15713, 2023

    arXiv 2023

  6. [6]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,

    N. Reimers, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," arXiv preprint arXiv:1908.10084, 2019

    arXiv 1908

  7. [7]

    PQuAD: A Persian question answering dataset,

    K. Darvishi, N. Shahbodaghkhan, Z. Abbasiantaeb, and S. Momtazi, "PQuAD: A Persian question answering dataset," Computer Speech & Language, vol. 80, p. 101486, 2023

    work page 2023

  8. [8]

    Searching for best practices in retrieval-augmented generation,

    X. Wang et al., "Searching for best practices in retrieval-augmented generation," arXiv preprint arXiv:2407.01219, 2024

    arXiv 2024

Show all 22 references
  1. [9]

    Benchmarking large language models in retrieval- augmented generation,

    J. Chen, H. Lin, X. Han, and L. Sun, "Benchmarking large language models in retrieval- augmented generation," in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, vol. 38, no. 16, pp. 17754-17762

    2024

  2. [10]

    Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models,

    Y. Lyu et al., "Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models," arXiv preprint arXiv:2401.17043, 2024

    2024 arXiv

  3. [11]

    Unsupervised cross-lingual representation learning at scale,

    A. Conneau, "Unsupervised cross-lingual representation learning at scale," arXiv preprint arXiv:1911.02116, 2019

    1911 arXiv

  4. [12]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, "Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters," in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 3505-3506

    2020

  5. [13]

    TookaBERT: A Step Forward for Persian NLU,

    M. SadraeiJavaheri et al., "TookaBERT: A Step Forward for Persian NLU," arXiv preprint arXiv:2407.16382, 2024

    2024 arXiv

  6. [14]

    AriaBERT: A Pre-trained Persian BERT Model for Natural Language Understanding,

    A. Ghafouri, M. A. Abbasi, and H. Naderi, "AriaBERT: A Pre-trained Persian BERT Model for Natural Language Understanding," 2023

    2023

  7. [15]

    Parsinlu: a suite of language understanding challenges for persian,

    D. Khashabi et al., "Parsinlu: a suite of language understanding challenges for persian," Transactions of the Association for Computational Linguistics, vol. 9, pp. 1147-1162, 2021

    2021

  8. [16]

    M. A. Mohammad Salar Mazyar, Mohammad Mahdi Azaryan, General Physical Education. University of Jiroft, 2018

    2018

  9. [17]

    Language-agnostic BERT Sentence Embedding,

    F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, "Language-agnostic BERT Sentence Embedding," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 878-891

    2022

  10. [18]

    Qwen2 technical report,

    A. Yang et al., "Qwen2 technical report," arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  11. [19]

    The Llama 3 Herd of Models,

    A. Dubey et al., "The Llama 3 Herd of Models," arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  12. [20]

    Gemma: Open models based on gemini research and technology,

    G. Team et al., "Gemma: Open models based on gemini research and technology," arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  13. [21]

    Gemma 2: Improving open language models at a practical size,

    G. Team et al., "Gemma 2: Improving open language models at a practical size," arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  14. [22]

    RAGAs: Automated Evaluation of Retrieval Augmented Generation,

    S. Es, J. James, L. E. Anke, and S. Schockaert, "RAGAs: Automated Evaluation of Retrieval Augmented Generation," in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2024, pp. 150-158

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.