{"id":"17c81075-bc3c-4a90-a7bd-e37bd92228bb","arxiv_id":"2501.04858","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Persian-focused embedding and language models are introduced and benchmarked for RAG, but evaluation inconsistencies prevent the main performance claims from being accepted.","lead":"The paper trains Persian-specific language and sentence-embedding models for retrieval-augmented question answering, then benchmarks them on general, scientific, and official-document datasets. It claims the new embeddings beat existing multilingual models, but its own tables contain inconsistencies and an unexplained competing model, which weakens the central result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5 cannot support the superiority claim: First Result Accuracy is defined as a percentage, yet PQuad reports 8,231 for MatinaSRoberta versus an 8,002-question test set; the Avg formula reproduces no value; and undeclared 'Ahd' beats it on Organizational Report.","rationale":"The paper's most important empirical assertion is that MatinaSRoberta outperforms existing embeddings for Persian retrieval. The only evidence for that assertion is Table 5, and the table is internally inconsistent with the paper's own metric definitions and dataset sizes. The reader's weakest assumption targeted exactly this inconsistency: Section 5.5.1 defines the retrieval metrics as percentages, but Table 5 reports values such as 8,231 for PQuad, which is larger than the 8,002-question test set described in Section 5.1.1. My independent check confirms that the weighted Avg formula cannot reproduce the reported values, and that an undeclared baseline ('Ahd') appears only in Table 5, with an Organizational Report row that beats MatinaSRoberta on Avg while being nearly identical to LaBSE. These are not stylistic issues; they make the central comparison uninterpretable. A second, compounding concern is train/evaluation overlap: Section 4 states that PQuad QA pairs were used in fine-tuning, and Section 5.1.1 describes a PQuad test split, but the paper never states that the test split was excluded from fine-tuning. If it was not excluded, any measured advantage on PQuad could reflect memorization rather than retrieval quality. Both concerns are empirical and could be settled by releasing the evaluation code, retrieval logs, and explicit data-split statements. Since the reader's REJECT verdict already reflects that the numerical evidence is not trustworthy, my stress-test does not change the verdict; it strengthens it by identifying an additional row-level contradiction that the reader did not emphasize.","tokens_in":12711,"tokens_out":4523,"duration_ms":40966,"concrete_test":"Request the retrieval logs and evaluation script behind Table 5. Recompute for each dataset and model: (i) first/second/third-result accuracy as percentages over the exact query set (PQuad test = 8,002; report the sizes of the scientific-specialized and organizational sets); (ii) total correct retrievals; (iii) Overall Score using the stated 3/2/1 weighting. Confirm all counts are <= query count, all percentages lie in [0,100], and state the Avg denominator. Then rerun the ranking. Also verify that the PQuad fine-tuning data described in Section 4 excludes the PQuad test split used in Section 5.1.1; if it does not, evaluate the retriever on a held-out set. If after this recomputation MatinaSRoberta ranks first on all three datasets and all values satisfy the definitions, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MatinaSRoberta is the best Persian retriever rests entirely on Table 5, but Table 5 cannot be read as stated. Section 5.5.1 defines First/Second/Third Result Accuracy as percentages of queries, yet the PQuad column reports 8,231 for MatinaSRoberta even though Section 5.1.1 fixes the PQuad test set at 8,002 queries. If these are raw correct-retrieval counts, they still cannot exceed the number of queries under a per-query 'first result' definition; if they are percentages, they are out of range. The weighted Overall Score formula (3*first + 2*second + third) does not reproduce any reported Avg value, and no denominator is specified. Independently, Table 5 introduces an 'Ahd' baseline that is never defined in Section 5.2, and for the Organizational Report MatinaSRoberta's Avg (21.02) is lower than Ahd's (22.52), while Ahd and LaBSE rows are nearly identical, contradicting Section 6.1's claim of consistent superiority across all three datasets. Without a corrected, code-released Table 5, the headline comparison is unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops Persian-specific language models for retrieval-augmented generation: MatinaRoberta (continual pretraining of XLM-RoBERTa Large on 73.11B Persian tokens) and MatinaSRoberta (Sentence-BERT-style fine-tuning of MatinaRoberta with multiple loss functions). It introduces three Persian RAG benchmarks (PQuad, a GPT-4-generated scientific-specialized dataset, and a GPT-4o-generated organizational-report dataset) and evaluates embedding models and LLMs under various RAG configurations. The headline claim is that the Persian-specialized embedding MatinaSRoberta consistently outperforms multilingual baselines across all three datasets in retrieval accuracy (Table 5), and that larger LLMs such as Llama-3.1-70B achieve the best generation quality (Tables 6, 10-12). The paper also reports ablation-style experiments on temperature tuning, chunk size, and document summary indexing.","tokens_in":13007,"tokens_out":2326,"duration_ms":21217,"significance":"If the central claims were fully established, the paper would make a useful practical contribution to Persian NLP and low-resource RAG: a new pretrained masked language model and sentence embedding model, benchmarks in three domains, and concrete configuration guidance (chunk size, temperature, summary indexing). The authors also provide a sizeable pretraining corpus description. However, the significance is currently capped by three unresolved issues: the central retrieval comparison in Table 5 is internally inconsistent with the stated metric definitions; the fine-tuning corpus overlaps with the evaluation data without any stated held-out split; and the scientific and organizational datasets are generated and evaluated by LLM-based pipelines without manual verification. Additionally, the paper does not release code, models, or datasets, which limits reproducibility of the quantitative claims. These issues must be resolved before the contributions can be assessed reliably.","major_comments":[{"comment":"The central superiority claim for MatinaSRoberta relies on Table 5, but that table cannot be interpreted under the metric definitions in Section 5.5.1. First Result Accuracy is defined as 'the percentage of queries where the correct answer was retrieved as the first result,' yet every PQUAD entry exceeds 100%: MatinaSRoberta reports 8,231, Ahd 7,231, LaBSE 4,713, L12-V2 4,087, Qwen2-7 3,979, Alibaba/gte-large 1,341, Alibaba/gte 814, while Section 5.1.1 fixes the PQuad test set at 8,002 questions. If these figures are raw correct-retrieval counts, they contradict the per-query 'first result' definition and still cannot exceed the number of test queries; if they are percentages, they are out of range. The analogous issue appears in the Scientific-Specialized and Organizational Report columns, where denominators are not specified anywhere. The weighted Avg formula (3*first + 2*second + third) also does not reproduce any reported Average: for example, MatinaSRoberta on PQuad would give (3*8231 + 2*475 + 142)/? = 47.70 only if divided by a denominator near 540, which is not stated. The paper must either correct the table to report interpretable percentages with explicit denominators, or provide the retrieval code and raw outputs so the reader can reproduce the ranking. As written, the headline comparison is unverifiable.","section":"Table 5 / Section 5.5.1"},{"comment":"Table 5 introduces a baseline named 'Ahd' that is never defined in Section 5.2 (which lists only MatinaSRoberta, LaBSE, L12-V2, Qwen2-7, Alibaba/gte-large, and Alibaba/gte). This is a load-bearing omission because Ahd outperforms MatinaSRoberta on the Organizational Report dataset on the Avg metric (22.52 vs. 21.02), directly contradicting the Section 6.1 statement that 'MatinaSRoberta model consistently demonstrated superior performance across all three datasets.' The paper must either identify what 'Ahd' is (architecture, checkpoint, training data) and include it in Section 5.2, or remove it from Table 5 and re-evaluate the consistency claim quantitatively. As presented, the table and the textual claim are mutually inconsistent.","section":"Table 5 / Section 5.2"},{"comment":"There is a circularity risk that is not addressed: Section 4 states that fine-tuning MatinaSRoberta used 'QA pairs from the PQuad collection,' while Section 5.1.1 and Table 5 use PQuad as the general-knowledge evaluation dataset. The paper never states whether the PQuad QA pairs used for fine-tuning were held out from the 8,002-question test set used in Table 5. If the same PQuad items (or their source Wikipedia passages) appear in both training and testing, the reported PQuad retrieval results are partly in-domain and the superiority claim is inflated. The authors must specify the exact split, the overlap-removal procedure between fine-tuning data and evaluation test, and re-run Table 5 on a genuinely held-out set if needed.","section":"Sections 4 and 5.1.1 / Table 5"},{"comment":"The scientific-specialized and organizational-report datasets are generated by GPT-4 and GPT-4o, respectively, and the RAGAS evaluation metrics are computed by LLM-based judges (faithfulness, answer relevancy, context precision, context recall). The paper provides no manual validation of the MCQ ground-truth answers, no human agreement study on the RAGAS judgments, and no reporting of the judge LLM used for RAGAS. Given that the same types of models are being evaluated, the possibility of systematic judge bias or dataset artifacts cannot be excluded. The authors should release the generated datasets, report a human-validated subset agreement rate for both MCQ correctness and RAGAS scores, and state which LLM served as the RAGAS judge. Without this, the generation-quality conclusions in Tables 10-12 and the optimization findings in Tables 7-9 rest on unverified auto-generated ground truth.","section":"Sections 5.1.2, 5.1.3, 5.4, Tables 10-12"},{"comment":"The RAGAS tables show that Context Precision is constant within each dataset (0.7750 for PQUAD, 0.8138 for Scientific-Specialized, and 0.5355 for two models in Organization Report), which is suspicious for a metric that is supposed to measure how well the retriever ranks relevant items for each query. The paper does not explain why the ranking quality is identical across all LLMs and embedding configurations. If Context Precision is computed from the same retrieved set for all models, then it is not a model-level metric as implied by the prose, or the reported values are rounded to four significant digits and coincidentally equal, which still requires explanation. The authors should clarify the computation pipeline and report the non-rounded values and any variance across evaluation runs.","section":"Tables 10-12"}],"minor_comments":[{"comment":"Table 1 lists token counts with a stated sum of 73.11B, but the rows sum to 73.11B only if 49.6 + 14.78 + 2.34 + 2.84 + 3.55 = 73.11; the values do sum correctly, yet the paper earlier in Section 3 says '54.69 billion Persian tokens,' which is inconsistent with the Table 1 sum of 73.11 billion. Please reconcile the corpus size.","section":"Section 3.1 / Table 1"},{"comment":"Table 3 is captioned 'Token Distribution Across Pretraining Datasets' but its rows describe fine-tuning hyperparameters for MatinaSRoberta (batch size, epochs, warmup ratio, etc.). The caption appears to be a copy-paste error and should be corrected.","section":"Table 3 title"},{"comment":"The Overall Score formula is described in words as weightings (3, 2, 1) for first/second/third result accuracy, but no explicit formula is given. Since Table 5's Avg column cannot be reproduced, the paper should provide the exact formula, including the denominator and how unanswerable questions are handled.","section":"Equation / Section 5.5.1"},{"comment":"Tables 7-9 report accuracy values without error bars or statistical tests, and the differences are often small (e.g., temperature 0 vs. 0.25 on PQUAD: 73.25 vs. 72.83). Given the sample sizes (8,002 PQuad questions), the paper should report confidence intervals or at least a paired significance test to support claims such as '0.25 provided the optimal balance' and 'smaller chunk size of 512 tokens produced the best overall results.'","section":"Tables 7-9"},{"comment":"The embedding model list includes 'Alibaba/gte' and 'Alibaba/gte-large,' but section 6.1 later refers to 'OpenAI embeddings' that are not listed in Section 5.2 or in Table 5. Please align the model descriptions with the actual evaluated models and remove the stray reference.","section":"Section 5.2"},{"comment":"The prose says 'MatinaSRoberta onsistently demonstrated the highest performance across most datasets' (typo 'onsistently'), and Section 6.1 claims superiority on the scientific-specialized dataset while Table 5 shows L12-V2 having a higher Avg (29.23) than MatinaSRoberta (42.23? no, 42.23 > 29.23, so this is fine only if the reader checks the numbers; the inconsistency with Ahd on Organizational Report remains as stated in the major comment). Please proofread and, more importantly, re-evaluate the 'consistent superiority' claim in light of the corrected Table 5.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has several load-bearing technical issues that prevent confirmation of its central claims: Table 5 is internally inconsistent with the metric definitions and contains an unexplained baseline; there is a potential train/eval overlap with PQuad; the auto-generated datasets and LLM-based evaluation lack manual validation; and the RAGAS tables show suspiciously constant context precision values. These issues are fixable in principle, so I do not recommend rejection, but they require substantial work: corrected and code-released retrieval metrics, explicit held-out splits, human validation of generated data, and clarification of the RAGAS computation. As a journal paper, the authors should also release the models and datasets to make the benchmarks reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. This is a resource-building paper for Persian RAG, and that work is real: MatinaRoberta and MatinaSRoberta are new trained artifacts, the 73B-token corpus is substantial, and the three-dataset benchmark covering general knowledge, scientific text, and formal documents is a sensible design. The authors also test temperature, chunk size, and document-summary indexing, which gives practitioners useful numbers even if the findings are predictable. I give credit where it is due.\n\nThe problem is that the central comparison does not hold up. Table 5, which is supposed to show that MatinaSRoberta 'consistently demonstrated superior performance across all three datasets,' is internally inconsistent. Section 5.5.1 defines First Result Accuracy as the percentage of queries where the correct answer was the first result, and the Overall Score as a weighted sum of the three rank accuracies. The PQuad column lists 8,231 for MatinaSRoberta, but PQuad has an 8,002-question test set. No percentage or per-query count can be larger than that. The Avg scores do not reproduce from the stated formula. An undeclared model named 'Ahd' appears in the table and beats MatinaSRoberta on the Organizational Report (22.52 vs. 21.02), contradicting the 'consistent' claim. The LaBSE and Ahd rows are nearly identical, suggesting a copy-paste error. This is not a cosmetic issue; it is the table the headline claim rests on.\n\nThere are also secondary problems. PQuad appears in fine-tuning (Section 4) and evaluation (Section 5.1.1), with no explicit statement that the test split was held out. None of the comparisons include error bars or significance tests. No code or data are released, so the models and benchmarks cannot be independently checked. The scientific and organizational datasets are GPT-generated and RAGAS-scored, which is fine if validated, but no human validation is reported.\n\nI do not think the paper is a waste. The models and benchmarks, if released with a corrected and reproducible evaluation, would be useful to the Persian NLP community. But as it stands, the numerical evidence does not support the advertised advantage. I would not cite this version, and I would not bring it to a reading group except to discuss what goes wrong when metric definitions and table numbers drift apart.","headline":"A substantial Persian RAG resource paper whose central superiority claim is undermined by an internally inconsistent Table 5; worth refereeing but not citable as-is.","tokens_in":13544,"tokens_out":5424,"would_cite":false,"duration_ms":49379,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Persian-specialized sentence-embedding model, created by continual pretraining and contrastive fine-tuning, retrieves relevant context for Persian question-answering better than generic multilingual embeddings…","keywords":["Retrieval-Augmented Generation","Persian NLP","Sentence Embeddings","Low-Resource Languages","Benchmarking","Language Model Fine-Tuning","Question Answering","Document Summary Indexing"],"falsifier":"Inspect the largest reported first-retrieval number for the general-knowledge set: if 'accuracy' means the percentage of queries, a value such as 8,231 cannot be correct for a test set described as 8,002 questions, so recomputing the table with the paper's stated weights and explicit denominators settles the claimed ranking.","tokens_in":12524,"feed_emoji":"🤖","tokens_out":11302,"duration_ms":94974,"temperature":0.7,"pith_summary":"Retrieval-augmented generation (RAG) for Persian is underdeveloped, and this paper sets out to close that gap with a Persian-specific embedding model and a benchmark. The authors build a masked language model on a 73.11-billion-token Persian corpus, fine-tune it into a sentence-embedding model, and evaluate both on three Persian datasets that cover general knowledge, scientific text, and formal policy documents. Their central claim is that the resulting embeddings retrieve more relevant context than multilingual embedding models, and that generation accuracy improves when retrieval is paired with tuned settings such as low temperature, smaller chunks, and document-summary indexing. If the claim holds, it gives practitioners a concrete recipe for building RAG systems in Persian and a template for other low-resource languages.","feed_headline":"Persian-trained embedding model beats multilingual retrieval rivals","feed_subtitle":"If the claim holds, low-resource languages gain a concrete recipe for better search and question answering.","key_machinery":"The load-bearing object is MatinaSRoberta, a Persian sentence-embedding model made in two stages. First, a multilingual masked language model is continually pretrained on 73.11 billion Persian tokens gathered from scientific articles, books, social media, websites, and a large web-crawl collection, creating the base encoder MatinaRoberta. Second, that encoder is fine-tuned with sentence-level objectives—Multiple Negatives Ranking Loss, Contrastive Loss, Softmax Loss, and Triplet Loss—over a mixed Persian and multilingual set of paired and triplet examples. The resulting dense 1024-dimensional embeddings are the mechanism that is supposed to capture Persian-specific morphology, flexible word order, and agglutination, which is what the paper argues gives it the retrieval advantage.","core_discovery":"The paper's discovery, stated on its own terms, is that training a sentence-embedding model on a large, native Persian corpus—rather than relying on multilingual embeddings—yields consistently better retrieval for Persian retrieval-augmented-generation (RAG) pipelines. The fine-tuned model uses 1024-dimensional embeddings produced by pooling token representations, trained with multiple ranking and contrastive losses on question-answer, entailment, paraphrase, and triplet data. The authors report that on the general-knowledge and scientific-specialized datasets this model ranks the correct context above multilingual baselines, and that on the formal organizational dataset it remains competitive. They also report that larger generator models produce the most accurate answers, and that retrieval-side choices—temperature, chunk size, and summary indexing—materially affect generation quality.","pith_inferences":["Ablating the two training stages would isolate how much of the retrieval gain comes from the Persian pretraining step versus the sentence fine-tuning step, which the paper does not test directly.","Comparing MatinaSRoberta against a multilingual sentence-embedding model fine-tuned with the same losses on the same pair data would show whether the advantage is language-specific or just an effect of extra fine-tuning.","Because the organizational-report evaluation uses a single policy document, the apparent formal-domain advantage may not generalize to other legal or bureaucratic Persian text until tested on a broader sample.","Applying the same data-and-fine-tuning recipe to a second low-resource language would be the most direct test of the paper's claim that its approach transfers."],"forward_implications":["Persian search engines and document-analysis tools can adopt these embeddings and expect more reliable retrieval than with multilingual defaults.","The two-stage recipe of continued pretraining on a native corpus followed by multi-loss sentence fine-tuning becomes a reusable template for other low-resource languages.","Practical RAG settings can be set with more confidence: a temperature near 0.25, a chunk size near 512 tokens for formal or technical text, and document summary indexing for long documents.","The three-dataset benchmark gives future Persian RAG models a standard frame for reporting retrieval and generation quality.","If the gains hold, smaller models fine-tuned this way could reduce the need to rely on very large generator models for low-resource RAG."],"supporting_citations":[{"why":"Supplies the general-knowledge Persian question-answering dataset that serves as one of the three evaluation benchmarks.","marker":"[7]"},{"why":"Is the multilingual masked language model that receives continued pretraining on Persian to form the base encoder.","marker":"[11]"},{"why":"Provides the sentence-embedding architecture and pooling approach on which the fine-tuning stage is based.","marker":"[6]"},{"why":"Provides the automated evaluation framework used for the generation-side metrics in the RAG assessment.","marker":"[22]"},{"why":"Supplies Persian language-understanding datasets used to fine-tune the embedding model on entailment and question-pair tasks.","marker":"[15]"}],"fun_headline_variants":["Persian-trained embeddings beat multilingual for RAG retrieval","MatinaSRoberta outperforms multilingual embeddings for Persian RAG","Custom Persian embedding model boosts RAG retrieval accuracy","Persian RAG gets native embedding edge over multilingual models","Native Persian embeddings improve retrieval for RAG systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the retrieval scores in Table 5 are computed as defined in Section 5.5.1, yet the reported numbers look like raw counts rather than percentages, so without knowing the normalization the model ranking is not interpretable.","fun_headline_variants_meta":{"raw":{"variants":["Persian-trained embeddings beat multilingual for RAG retrieval","MatinaSRoberta outperforms multilingual embeddings for Persian RAG","Custom Persian embedding model boosts RAG retrieval accuracy","Persian RAG gets native embedding edge over multilingual models","Native Persian embeddings improve retrieval for RAG systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1818,"prompt_tokens":946,"completion_tokens":872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":562,"tokens_out":872,"duration_ms":6284,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:47.838205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the largest reported first-retrieval number for the general-knowledge set: if 'accuracy' means the percentage of queries, a value such as 8,231 cannot be correct for a test set described as 8,002 questions, so recomputing the table with the paper's stated weights and explicit denominators settles the claimed ranking.","supporting_citations":[{"cited_title":"PQuAD: A Persian question answering dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the general-knowledge Persian question-answering dataset that serves as one of the three evaluation benchmarks."},{"cited_title":"RAGAs: Automated Evaluation of Retrieval Augmented Generation,","cited_arxiv_id":null,"evidence_quote":"Provides the automated evaluation framework used for the generation-side metrics in the RAG assessment."},{"cited_title":"Parsinlu: a suite of language understanding challenges for persian,","cited_arxiv_id":null,"evidence_quote":"Supplies Persian language-understanding datasets used to fine-tune the embedding model on entailment and question-pair tasks."}],"review_version":1}