REVIEW 4 major objections 4 minor 2 references
Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A 1B embedder fine-tuned on 65,773 synthetic Greek retrieval pairs lifts ranking quality from 0.362 to 0.835 on specialist domains, beating both BM25 and an 8B multilingual embedder—but the advantage over BM25 reverses on general-domain…
desk verdict Useful, honest Greek RAG paper whose headline numbers rest on synthetic queries; the benchmark and the BM25-vs-dense measurement are the real contributions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is contrastive fine-tuning with hard negatives: a Nemotron-3-Embed-1B model is trained with InfoNCE, pulling each synthetic query toward its gold passage and away from in-batch positives and seven mined hard negatives per query, with GradCache (a technique that decouples logical batch size from memory) allowing a logical batch of 256 at 4,096 tokens so the gradient sees many negatives. The negatives are mined from an 8B multilingual embedder's cosine-kNN window, with the top ranks and near-duplicates excluded, and the queries themselves are synthesized by instruction models few-shot conditioned on real user queries and audited for grounding. The comparison axis is a Greek-aware BM25 baseline, which exposes that off-the-shelf dense models lack Greek exposure; the reader is a LoRA-tuned hybrid Mamba-Transformer mixture-of-experts model trained on cite-and-abstain examples with shuffled gold positions and evaluated on HERA, a 4,946-item Greek benchmark with 25% unanswerable items.
What would settle it
Take a sample of real Greek user queries from production, have bilingual annotators judge relevance on the same specialist corpora, and score the adapted 1B embedder against BM25; the in-domain claim predicts a positive gap near +0.080 nDCG@10, while the paper's own unpublished observation predicts roughly +0.04, and a zero or negative gap would show the synthetic evaluation overstates the adapted model's advantage.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that language exposure, not model capacity, is the binding constraint for Greek retrieval. A 1B embedder contrastively fine-tuned on 65,773 bilingual Greek/English retrieval pairs raises normalized discounted cumulative gain at 10 (nDCG@10) from 0.362 to 0.835 on specialist domains, outperforming both a parameter-free BM25 baseline (0.757) and the 8B Qwen3-Embedding model (0.680) that beats it off the shelf. The learned competence transfers: on general-domain Greek Wikipedia, the adapted 1B beats its unadapted base by +0.399, though it loses to BM25 by +0.081 there. The paper also shows that a cross-encoder reranker only pays off after adaptation, and that a LoRA-tuned 30B-A3B mixture-of-experts reader nearly triples faithfulness and more than doubles judged correctness, while abstention on unanswerable questions remains largely unsolved.
Load-bearing premise
The load-bearing premise is that LLM-generated synthetic queries, written from the gold passages and few-shot conditioned on real user queries, behave like real Greek user queries well enough to train and evaluate on, since no human relevance labels exist anywhere in the corpus.
Editorial extensions
If this is right
- Any Greek RAG pipeline that replaced BM25 with an off-the-shelf multilingual embedder made retrieval worse on the specialist domains tested; the evidence-backed configuration is BM25 plus an adapted dense retriever fused by reciprocal rank fusion, which beats the better component in both in-domain and out-of-domain regimes.
- Scale is not the fix: within the off-the-shelf family, 8B and 4B are statistically indistinguishable in-domain, while a 1B model adapted on Greek gains over +0.42 nDCG@10 on every domain, showing exposure to the language matters more than capacity.
- The learned Greek competence transfers out of domain, but the win over BM25 is domain-dependent and reverses on general Greek Wikipedia, so deployments on general Greek should expect to lose to a lexical baseline.
- An adapted cross-encoder reranker is worth +0.061 over the no-rerank floor end to end, while an off-the-shelf reranker's contribution is inconsistent across samples; no evaluation without the no-rerank floor can distinguish the two.
- The LoRA-tuned reader more than doubles judged answer correctness and nearly triples faithfulness, but abstention on unanswerable questions improves to only 30.5%, so refusing to answer remains unsolved in this pipeline.
Reading between the lines
- If real Greek user queries differ from the synthetic ones as much as the paper's own unpublished production observation suggests—about half the synthetic margin over BM25—then the in-domain retrieval numbers are upper bounds on production lift, not measurements of it.
- The default max_len of 512 truncating 87% of training pairs is likely a general failure mode for long-document languages; other adapters on Greek or similarly formal, jargon-dense languages may silently train on truncated positives.
- The BM25-versus-dense ordering on specialist Greek may extend to other low-resource languages with formal text and scarce native instruction data; a cheap test is to run the same contrastive adaptation on another language and compare against BM25.
- A logged, sized human audit of HERA items would settle how much of the judged reader gains are real; the paper's own unsized spot-check and model-family judge confound leave that open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts NVIDIA's Nemotron retrieval stack to Modern Greek end to end: it mines a 65,773-query specialist corpus, synthesizes queries from gold chunks, fine-tunes a 1B embedder and a reranker, LoRA-tunes a 30B-A3B MoE reader, and introduces a new Greek RAG benchmark called HERA. The main empirical claims are that BM25 beats several off-the-shelf multilingual dense embedders on specialist Greek domains, that contrastive fine-tuning raises the 1B embedder's nDCG@10 from 0.362 to 0.835 in-domain, that an adapted reranker reliably adds value, and that LoRA tuning raises judged grounded answer correctness from 29.4% to 66.9%. The paper is unusually candid about its limitations, including the synthetic nature of all retrieval queries, an unpublished production observation that undercuts the synthetic margin, a judged-reader confound, and two claims that a larger second evaluation did not reproduce.
Significance. If the central retrieval ordering survives contact with real Greek user queries, the paper is a valuable contribution: it ships adapted models and a new benchmark, reports paired bootstrap intervals for every retrieval comparison, and demonstrates that a parameter-free lexical baseline remains competitive for a language that is out of distribution for current multilingual embedders. The paper's honesty about its own failed claims and re-evaluations is a genuine strength, as is its explicit separation of in-domain and out-of-domain behavior. However, the load-bearing retrieval results are measured entirely on LLM-generated queries that echo their gold passages, and the paper's own unpublished production observation suggests that the margin over BM25 roughly halves on real queries. The judged reader numbers are similarly exposed to a same-family LLM judge. These issues do not invalidate the artifacts or the relative ordering as a hypothesis, but they mean the headline magnitudes and the production-oriented recommendations are not yet established.
major comments (4)
- [3; Tables 2-4] The central retrieval evaluation rests on synthetic queries generated from the gold chunks, and the paper explicitly concedes that synthesized queries echo their source passage's vocabulary and that on an internal production sample the margin over BM25 was roughly +0.04 rather than +0.080. This is a load-bearing issue for Section 5's practical claim that Greek RAG systems should not replace BM25 with an off-the-shelf dense model and for Section 6's headline +0.080 margin. The internal observation cannot serve as evidence because no query count or labelling protocol is attached. I ask for a real-query, human-labelled evaluation with explicit annotation guidelines and inter-annotator agreement, covering at least BM25, the adapted 1B embedder, an off-the-shelf dense model, and the RRF fusion. If that evaluation is not feasible, the production-scope statements should be explicitly reframed as hypotheses about real traffic rather than measured results.
- [4; Section 9] The reader judge is a served Qwen3.5-122B-A10B-FP8, the same model family that generated HERA's items, and judged answer correctness is scored against a model-authored reference answer. The paper acknowledges the confound but does not measure it, and the statement that faithfulness, citation, and abstention metrics are anchored to the benchmark's structure does not cover the correctness number, which is the headline reader claim (29.4% to 66.9%). Because the reader numbers are also single-run point estimates with no intervals, the headline correctness gain should be re-scored with an independent-family judge or a logged human evaluation, or it should be reported with an explicit caveat that it is an upper bound rather than a measured effect.
- [3; Section 4] The labelling bet that the chunk a query was written from is that query's positive is the basis for both training and evaluation, and no human relevance marks exist anywhere in the corpus. The grounding audit checks that query specifics appear in the gold chunk, but it does not establish that the gold chunk is the only relevant document or that the synthetic query distribution matches real user queries. This closed loop inflates both the adapted model's scores and BM25's absolute standing. The paper's own production observation is the only direct evidence about the loop's severity, and it is unpublished. I request a small human-relevance-judged sample drawn from the production query distribution so that the label bet can be tested rather than assumed.
- [Availability; Section 4] The retrieval training corpus is withheld, and the HERA human pass is an unsized spot-check with the number of checked items not logged. These two gaps limit reproducibility and verification of the central claims. The paper says retrieval results can be reproduced against released models and the benchmark but not retrained from scratch; for a paper whose central method is adaptation, the training corpus (or a sufficiently large licensed subsample) should be released, or a formal reproducibility protocol should be provided. In addition, the number of HERA items that were hand-checked should be reported, since Section 9's judged correctness numbers depend on the benchmark's validity.
minor comments (4)
- [4; References] The construction models Qwen3.5-122B and GLM-5.2 are central to HERA's two-stage verification but are not included in the reference list; please add citations or technical reports for both.
- [3] The internal production comparison between real and synthetic queries is reported as '+0.04' versus '+0.080' with no sample size or confidence interval. If this observation is retained, it should be accompanied by the query count and, ideally, a paired bootstrap interval; otherwise, it reads as an anecdote that the paper itself describes as unpublished.
- [Table 2; Appendix A] The abstract and Section 5 quote macro-averaged nDCG@10 values, but Table 2 reports both macro-over-sets and macro-over-queries, which differ non-trivially (0.8352 vs. 0.8575). Please state explicitly in the main text which weighting is used for each headline number to avoid reader confusion.
- [10] The 'four ways our instruments lied' section is valuable, but the fourth item (distributed loss summed rather than averaged) is not tied to any reported number or figure; please state which runs, if any, were affected by this bug and how the reported results were verified against it.
Circularity Check
No circular derivation found: the paper's central retrieval and reader claims are empirical measurements on held-out data, with the synthetic-query and same-family-judge loops explicitly declared as limitations rather than hidden reductions.
full rationale
The paper's derivation chain is empirical rather than circular. Queries are synthesized from gold chunks and labeled with the source chunk as positive, and the same synthetic distribution is used for training and held-out evaluation; however, the held-out queries are unseen, and the paper explicitly states that this makes retrieval numbers an upper bound on production behavior, not a prediction from the fitted data. The same-family judge-generator confound in HERA is declared in Section 4 ('A confound we have to declare'), with the paper separating exposed correctness from structure-anchored faithfulness and abstention metrics. There is no load-bearing self-citation: the reference list contains no prior work by the present authors, and the teacher model Sophea-Titan-1 is released as an artifact rather than cited as proof. The paper even reports two claims that a second evaluation did not reproduce and four instrument failures, demonstrating self-correction rather than circular reinforcement. The validity threats identified—synthetic queries overstating production lift and judge family preference—are real limitations on the strength of the conclusions, but no equation or claim reduces to its own input by construction. Therefore, no circularity step is exhibited, and the appropriate finding is a low score reflecting declared caveats rather than circular derivation.
Assumptions & free parameters
assumptions (5)
- ad hoc to paper The chunk a query was written from is the query's positive; no human relevance labelling exists in the corpus.
- domain assumption Instruct-model-generated queries, few-shot conditioned on real production queries, are a valid proxy for real user queries for ranking system comparison.
- domain assumption The LLM judge from the Qwen3.5 family provides trustworthy correctness and faithfulness scores on HERA despite the family overlap with HERA's generator.
- standard math Greek-aware BM25 with standard k1=1.2 and b=0.75 is an appropriate lexical baseline.
- domain assumption HERA items are valid because Qwen3.5-122B generation plus GLM-5.2 verification plus an unsized human spot-check suffice.
Cite this review
Pith. "Pith review of Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains." pith.science (2026). https://pith.science/paper/RMDYESAD
@misc{pith2026260805138,
author = {Pith},
title = {Pith review of: Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains},
year = {2026},
howpublished = {\url{https://pith.science/paper/RMDYESAD}},
note = {Machine review of arXiv:2608.05138}
}
read the original abstract
Modern Greek is absent from NVIDIA's Nemotron retrieval models and from major multilingual retrieval benchmarks, despite being important for retrieval-augmented generation (RAG) in legal, energy, financial, and medical applications. We present an end-to-end adaptation of the Nemotron retrieval stack for Modern Greek, including corpus mining, synthetic supervision, retrieval model training, reranker adaptation, reader fine-tuning, and a new benchmark called HERA. Our study shows that a parameter-free BM25 baseline outperforms several off-the-shelf multilingual dense retrieval models on specialist Greek corpora. After fine-tuning on 65,773 Greek retrieval pairs, a Nemotron 1B embedder improves nDCG@10 from 0.362 to 0.835 and substantially outperforms its unadapted counterpart. The learned language competence transfers to general-domain Greek, although the advantage over BM25 remains domain-dependent. We further adapt a cross-encoder reranker and demonstrate consistent improvements across specialist domains. Finally, we LoRA-tune a Nemotron 30B-A3B mixture-of-experts reader for grounded generation, increasing judged answer correctness from 29.4% to 66.9% while significantly improving faithfulness and citation quality. We also introduce HERA, the first large-scale Greek benchmark for retrieval-augmented generation, and release our adapted models and benchmark to support future research on Greek-language RAG systems.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[2009]
In Proceedings of SIGIR 2009 , pages 758–759
Reciprocal Rank Fusion Outperforms Condorcet and Indi - vidual Rank Learning Methods. In Proceedings of SIGIR 2009 , pages 758–759. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. RAGAS: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of EACL 2024 (System Demonstrations). William Fedus, Barret Zoph, and Noa...
arXiv 2009
-
[2023]
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track. A Metric definitions Ranking quality. For a ranked list of 𝑘 documents with binary relevance 𝑟𝑖 ∈ {0, 1}, discounted cumulative gain and its normalised form are (Järvelin and Kekäläinen, 2002) DCG@𝑘 = ∑ 𝑘 𝑖=1 𝑟𝑖 log2(𝑖 + 1), nDCG@𝑘 = DCG@ 𝑘 IDCG@𝑘 where IDCG@...
work page 2002
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.