Pith. sign in

REVIEW 3 major objections 5 minor 66 references

TR-RAG couples on-policy rewards with a frozen-teacher reverse-KL on student prefixes, improving language adherence and evidence use when answers must be non-English but evidence is English.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:46 UTC pith:J5RHQWZI

load-bearing objection Clean joint recipe for a real industrial failure mode: on-policy reverse-KL as a safety net against reward-only language collapse under English evidence. the 3 major comments →

arxiv 2607.02966 v2 pith:J5RHQWZI submitted 2026-07-03 cs.CL

Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG

Haotian Zhou , Weiran Huang , Siqi Liu , Xiting Wang , Xin Zhang , Zhihao Wen This is my paper
classification cs.CL
keywords cross-lingual RAGEnglish-evidence generationon-policy distillationteacher-regularized RLlanguage driftprefix mismatchcharacter n-gram reward
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When users query in many languages but the knowledge base stays English, generators often drift into English or code-switching and use the passages unreliably. The paper traces this to two training problems: errors depend on the prefixes the model actually visits, so fixed-trajectory supervision never corrects them, and sequence-level rewards give noisy credit. TR-RAG answers both by letting a compact student sample answers on-policy while a stronger frozen teacher is queried only on those same prefixes, supplying a reverse-KL anchor that is optimized jointly with a three-part reward for language consistency, character 3-gram recall, and judge-scored groundedness. Across biomedical, multi-hop, and native multilingual benchmarks the method raises the joint score of language adherence and evidence-grounded correctness; the teacher anchor prevents the large language collapses pure reward RL can suffer and still lifts grounding on distant languages where reward-only training stalls. Anyone building cross-border assistants or global support systems cares because small models can stay fluent and faithful without non-English knowledge bases.

Core claim

TR-RAG attains the best composite of language adherence and evidence-grounded correctness across BioASQ-ENKB5, Hotpot-ENKB5 and MKQA with two compact backbones. The frozen-teacher reverse-KL anchor on student-visited prefixes acts as a safety net: on in-domain languages it prevents large language-consistency collapses (up to about 27 percentage points) that reward-only RL can suffer by drifting below the base model, while on distant out-of-distribution languages, where reward-only RL stalls at the base ceiling, it still improves evidence grounding; on character 3-gram recall the compact student sometimes surpasses its 70B teacher.

What carries the argument

The prefix-wise reverse-KL teacher anchor: the student samples on-policy answers; a stronger frozen teacher is queried only on those student-visited prefixes (no teacher rollouts) and supplies the length-normalized student-to-teacher reverse KL, which is combined with a three-component sequence reward (language consistency, character 3-gram recall, LLM-judge groundedness) inside a single GRPO-style objective.

Load-bearing premise

The method assumes that a fixed mix of language-ID, character-overlap and a single LLM-judge score, plus a modest reverse-KL weight, is a faithful enough proxy for true multilingual generation quality that optimizing it yields better real answers.

What would settle it

On a held-out native multilingual English-evidence RAG set scored by humans, if reward-only RL matched or beat TR-RAG on both language consistency and human-rated faithfulness without large language collapses, the safety-net claim for the teacher anchor would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Compact generators can be post-trained for English-evidence cross-lingual RAG without non-English knowledge bases or teacher rollouts.
  • Pure reward-only RL is unsafe for language control: the teacher anchor is required to avoid large in-domain language-consistency collapses.
  • On distant low-resource languages the same anchor still lifts evidence grounding even when language consistency is capacity-bottlenecked at the base model.
  • Character 3-gram recall is a dense, tokenization-robust signal that can let a small student exceed a much larger teacher on lexical content transfer.
  • Combining on-policy distillation with task rewards strictly dominates either alone on the composite metric.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same student-visited-prefix reverse-KL recipe may stabilize other early-error cascade settings such as code-switching, mixed-language evidence, or multimodal RAG.
  • If language consistency stays weak on extreme scripts, richer script-aware rewards could raise the OOD ceiling without changing the anchor.
  • Systems that already hold a large same-family teacher can treat the anchor as a cheap safety layer rather than a full capability-transfer pipeline.
  • The insurance-premium framing suggests a general pattern: accept a small judge-score tax in safe regimes to buy protection against catastrophic language drift.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies English-evidence cross-lingual RAG, where queries and answers are non-English but retrieved passages remain English. It attributes generation failures (language drift and brittle evidence use) to prefix mismatch under fixed-trajectory supervision and high-variance credit assignment under sequence-level rewards. TR-RAG couples GRPO-style reward optimization with a length-normalized reverse-KL anchor from a frozen teacher queried only on student-visited prefixes (Eqs. 9–11), using a three-component reward of language consistency, character 3-gram recall, and an LLM-judge score (Eq. 3). Across BioASQ-ENKB5, Hotpot-ENKB5, and native MKQA, with Llama-3B and Qwen-4B students, TR-RAG reports the best composite of language adherence and evidence-grounded correctness, and the teacher anchor is claimed to act as a safety net against large in-domain LC collapses (up to ~27 pp) that reward-only RL can suffer while still improving grounding on distant OOD languages.

Significance. If the results hold, the paper offers a practical, generation-centric post-training recipe for a recurring industrial setting (English-only knowledge bases with multilingual users). Strengths include a clear problem formulation, a clean on-policy distillation design that avoids teacher rollouts, multi-benchmark and multi-backbone evaluation, full ablations of RL-only / OPD-only / reward components (Table 2), β sensitivity, weak-teacher controls, cross-family judges, and qualitative cases. The safety-net claim is concrete and falsifiable (e.g., Hotpot Qwen-4B Indonesian LC 28.27% for Naive-RL vs. 55.60% for TR-RAG in Table 11). The work is a solid systems contribution rather than a foundational theoretical advance, but it is well-scoped and useful for the community.

major comments (3)
  1. The central evaluation relies on a composite that includes an LLM-judge which is also a training reward component (§3.2, Eq. 3; Metrics in §4.1). Although the paper reports cross-family Llama judges (Table 7 / §D.2.4) and judge-independent LC and char-3-gram signals, the main tables still lead with the same-family Qwen judge for ENKB-RAG-5. For the composite claim to be fully load-bearing, the main results should foreground the judge-independent metrics and the cross-family judge as primary, not only as appendix checks, so that the ranking is not partly driven by the training reward itself.
  2. The ~27 pp safety-net claim is supported most strongly by a single setting (Hotpot-ENKB5, Qwen-4B, Indonesian LC in Table 11: Naive-RL 28.27% vs. TR-RAG 55.60%). Elsewhere LC is often near-saturated or subject to acknowledged LID artifacts (§D.5). The paper should either (i) quantify how often and under what conditions reward-only RL collapses across all backbone–dataset pairs, or (ii) reframe the safety-net claim as conditional on settings where unconstrained RL is unstable, rather than as a general property of the method.
  3. ENKB-RAG-5 is constructed by machine-translating questions and short answers while leaving English evidence untouched (§4.1, Appendix D.1). The authors acknowledge translationese and residual term drift as limitations, and MKQA is a helpful native check, but the main training and most of the reported gains still rest on the translated construction. A clearer quantification of how much of the composite gain survives when both training and evaluation are restricted to native multilingual queries (or a stronger statement that the method is intended primarily for the controlled English-evidence construction) would strengthen the external-validity claim.
minor comments (5)
  1. Single-seed reporting is acknowledged (§D.4.5) but should be stated more prominently in the main experimental section; variance over a few seeds would help readers assess the magnitude of the composite gaps.
  2. The Hotpot LC plateau and fastText LID artifacts (§D.5) make the LC column hard to interpret for several languages; consider scoring against the known query language ℓ(q) as the primary LC metric in the main tables.
  3. Notation: M vs. K for number of passages vs. rollouts is explained in the figure caption but could be stated once in §2 for clarity.
  4. Appendix B is thorough; a short pointer in the main text to which lemmas are used for the reverse-KL estimator and length-normalization would help readers who skip the appendix.
  5. Figures 3–7 and 17–21 are useful; ensure axis labels and β units are consistent across the main text and appendix versions.

Circularity Check

1 steps flagged

No significant circularity: empirical RL recipe evaluated on held-out languages and native MKQA; dual-use of LLM-judge is a mild validity caveat, not a by-construction derivation.

specific steps
  1. other [§3.2 Eq. (3) reward; §4.1 Metrics / Composite; App. D.2 dual-use discussion]
    "r(q, D, y) = λ_lang r_lang(y, a⋆) + λ_3g r_3g(y, a⋆) + λ_judge r_judge(q, D, a⋆, y) ... Composite = 1/3(r_lang + r_3g + r_judge). ... We use an LLM-judge ... in two roles: (i) as a training reward component r_judge ... and (ii) as one of the evaluation metrics"

    Training optimizes the same three signals that define the reported Composite, and the primary judge family overlaps the training reward judge. This is train–eval alignment / dual-use, not a fitted parameter renamed as a prediction or a self-definitional identity. Mitigated by cross-family Llama judge, judge-independent LC and char-3-gram, and ablations; does not force the safety-net LC result.

full rationale

TR-RAG is an empirical post-training recipe, not a first-principles derivation. The load-bearing chain is: (i) English-evidence generation fails via prefix-dependent language drift and brittle evidence use; (ii) fixed-trajectory SFT/KD suffers prefix mismatch while sequence-level rewards give high-variance credit assignment; (iii) couple GRPO-style reward optimization with a frozen-teacher reverse-KL on student-visited prefixes (Eqs. 9–11); (iv) report composite gains and a safety-net effect on language consistency. None of these steps reduce by definition to their inputs. The reverse-KL form and on-policy prefix querying follow external GKD/OPD literature, not a self-authored uniqueness theorem. Reward components (LID, char-3-gram, LLM-judge) are design choices, not fitted parameters renamed as predictions. Composite is the mean of the three evaluation signals that also appear in the reward, which is standard train–eval alignment rather than circular derivation: baselines are scored on the same metrics, ablations show RL-only and OPD-only underperform the combination, and the headline safety-net claim (up to ~27 pp in-domain LC collapse under Naive-RL) is visible on the judge-independent LID metric (e.g., Hotpot Qwen-4B Indonesian LC 28.27% vs 55.60%). Residual dual-use of a Qwen-family judge in both reward and primary evaluation is a mild self-preference risk, already partially controlled by cross-family Llama judges, char-3-gram, and LC; it does not make the central claim true by construction. Score 1 for that minor dual-use only.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard policy-gradient and KL identities plus several engineering choices (reward weights, β, LID threshold, translation pipeline) that are fixed by hand or small grid search rather than derived. No new physical or mathematical entities are postulated; the invented objects are the method name and the reward decomposition itself.

free parameters (4)
  • teacher-anchor weight β = 0.02
    Selected by grid search on a held-out split (β∈{0.0002,0.002,0.02,0.2}); fixed at 0.02 for all main runs. Directly controls the strength of the safety-net claim.
  • reward weights λ_lang, λ_3g, λ_judge = 0.5 / 1 / 1
    Hand-chosen (0.5, 1, 1) after a small sanity check; λ_lang lowered to avoid early language dominance. Scale the composite objective that is optimized and reported.
  • GRPO group size K and advantage ε = K=8, ε=1e-6
    K=8 rollouts, ε=1e-6; standard but free choices that affect variance of the policy-gradient term.
  • length-penalty threshold and LID short-text cutoff = 3500 chars / 10 chars
    Answers longer than 3500 characters are penalized; LID skipped below 10 characters. Implementation details that alter the realized reward surface.
axioms (5)
  • standard math Policy-gradient / GRPO advantage estimation with group-mean baseline is an unbiased (or low-bias) estimator of the expected reward gradient.
    Invoked for the reward term in §3.3 and Algorithm 1; standard RL assumption.
  • domain assumption Length-normalized reverse KL on student prefixes is a valid local trust-region regularizer that does not require teacher rollouts.
    Core of the teacher anchor (Eq. 9–11); justified by GKD-style arguments and Appendix B lemmas but still an algorithmic modeling choice.
  • domain assumption fastText lid.176 correctly identifies the language of free-form answers for the languages studied (with the short-text exception).
    Used both as training reward and evaluation metric; Hotpot LC plateau is acknowledged as a LID artifact in Appendix D.5.
  • domain assumption The LLM-as-judge score (Qwen3-Next-80B primary) is a sufficiently faithful continuous proxy for evidence-grounded semantic correctness.
    Enters the reward (Eq. 6) and evaluation; partially validated by cross-size and cross-family judges.
  • ad hoc to paper Machine-translated questions/answers (ENKB-RAG-5) preserve the generation-side failure modes of native English-evidence cross-lingual RAG.
    Dataset construction premise in §4.1 and Appendix D.1; mitigated by native MKQA evaluation but still load-bearing for the main tables.
invented entities (2)
  • TR-RAG joint objective (reward + prefix-wise reverse-KL) no independent evidence
    purpose: Couples sequence-level multilingual rewards with dense on-policy teacher guidance to address prefix mismatch and high-variance credit assignment.
    The named recipe is the paper’s central technical object; independent evidence is the empirical tables, not an external physical prediction.
  • Three-component English-evidence multilingual reward (LC + char-3-gram + LLM-judge) no independent evidence
    purpose: Provide complementary training signals for language adherence, lexical faithfulness and evidence-grounded correctness.
    Designed for this setting; components individually known but the weighted combination is paper-specific.

pith-pipeline@v1.1.0-grok45 · 50477 in / 3648 out tokens · 32186 ms · 2026-07-12T05:46:14.666712+00:00 · methodology

0 comments
read the original abstract

Cross-lingual retrieval-augmented generation (RAG) is often deployed in an English-evidence regime, where users query in diverse languages but retrieved passages remain English. In this setting, generation can fail despite strong base models: English evidence induces language drift (English or code-switching outputs) and models use evidence unreliably when producing non-English answers. We attribute these failures to two post-training challenges: (i) errors are prefix-dependent, so fixed-trajectory supervision suffers from prefix mismatch; and (ii) sequence-level (partly discrete / judge-based) rewards yield noisy credit assignment and high-variance updates. We propose TR-RAG, a teacher-regularized RL recipe that couples reward optimization with on-policy distillation on student-visited prefixes. A compact student samples on-policy answers, while a stronger frozen teacher is queried only on those prefixes and provides a prefix-wise student-to-teacher reverse-KL anchor. We further introduce a reward decomposition for English-evidence multilingual generation, combining language consistency, character 3-gram recall, and an LLM-judge score for evidence-grounded correctness. Across three benchmarks (BioASQ-ENKB5, Hotpot-ENKB5, and naturally multilingual MKQA) and two backbones, TR-RAG improves the composite of language adherence and evidence-grounded correctness over strong baselines. Crucially, the teacher anchor acts as a safety net: on in-domain languages it prevents the large language-consistency collapses (up to ~27 percentage points) that reward-only RL can suffer by drifting below even the base model, while on distant out-of-distribution languages, where reward-only RL stalls at the base model's ceiling, it still improves evidence grounding; and on character 3-gram recall the compact student sometimes surpasses its 70B teacher.

Figures

Figures reproduced from arXiv: 2607.02966 by Haotian Zhou, Siqi Liu, Weiran Huang, Xin Zhang, Xiting Wang, Zhihao Wen.

Figure 1
Figure 1. Figure 1: Overview. (a) English-evidence cross-lingual RAG: queries/answers are in a target language while retrieved passages are English. (The panel labels the retrieved documents “Top-K” in the generic retrieval sense; throughout the text their count is denoted M, as K is reserved for the number of on-policy rollouts per input.) (b) Two prefix-dependent failure modes (language drift and evidence￾use failures) indu… view at source ↗
Figure 2
Figure 2. Figure 2: Framework. (a) Inputs: a multilingual query q and English-only evidence D. (b) The student samples on-policy rollouts and receives the task reward. (c) The teacher is queried only on student prefixes to compute a KL anchor (no teacher rollouts). (d) The student is updated with policy gradients and a frozen-teacher reverse-KL anchor. Student rollouts (on-policy trajectories). (Fig. 2b) For each training ins… view at source ↗
Figure 5
Figure 5. Figure 5: Training time (Qwen-4B backbone, BioASQ + Hotpot). TR-RAG ≈ Naive￾RL/OPD (compact/MoE teacher). 0 100 200 Steps 1.2 1.4 1.6 1.8 Reward [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative case study (Llama-3.2-3B-Instruct backbone, Indonesian, Hotpot-ENKB5). Highlighting is consistent across panels: red marks language drift (Indonesian slipping into copied English spans) or evidence-unsupported content, and green marks correct, evidence-grounded content in the target language. Baseline systems drift and latch onto a salient but unsupported entity, whereas TR-RAG stays in fluent … view at source ↗
Figure 9
Figure 9. Figure 9: Indonesian translation prompt. Prompt template used to translate English queries into Indonesian (id) when constructing BioASQ-ENKB5 and Hotpot-ENKB5. D.2.1 On the inadequacy of exact match for free-form cross-lingual RAG. A natural question is why we do not report Exact Match (EM) against the gold answer. The reason is that our setup is open-domain, free-form generation: instruction-tuned student models a… view at source ↗
Figure 10
Figure 10. Figure 10: Japanese translation prompt. Prompt template used to translate English queries into Japanese (ja) when constructing BioASQ-ENKB5 and Hotpot-ENKB5. D.2.2 LLM-judge reliability and potential overfitting. We use an LLM-judge (Qwen3-Next-80B-A3B-Instruct) in two roles: (i) as a training reward component rjudge to encourage evidence-grounded correctness, and (ii) as one of the evaluation metrics reported in th… view at source ↗
Figure 11
Figure 11. Figure 11: Korean translation prompt. Prompt template used to translate English queries into Korean (ko) when constructing BioASQ-ENKB5 and Hotpot-ENKB5 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Thai translation prompt. Prompt template used to translate English queries into Thai (th) when constructing BioASQ-ENKB5 and Hotpot-ENKB5. Removing the conciseness criterion lifts the judge score by +1.85 (Qwen) and +1.35 (Llama), confirming that the small 3-gram–judge gap is largely a length-conciseness artifact rather than a correctness/faithfulness regression. D.2.4 Cross-family judge: full comparison.… view at source ↗
Figure 13
Figure 13. Figure 13: Vietnamese translation prompt. Prompt template used to translate English queries into Vietnamese (vi) when constructing BioASQ-ENKB5 and Hotpot-ENKB5. D.3 Baselines The description for baselines is shown below: • Translate. This baseline reduces the cross-lingual generation problem to a pivot-English pipeline. Specifically, given a user query q in language ℓ, we first translate it into English (qˆen), con… view at source ↗
Figure 14
Figure 14. Figure 14: LLM-judge prompt. We use a rubric-based evaluator prompt to score student answers for RAG, conditioned on the question q, retrieved English context D, reference answer a ⋆ (training only), and student output y. The prompt instructs the judge to assess (i) faithfulness (hallucination checking against D), (ii) correctness (alignment with a ⋆ , with entity focus for BioASQ and multi￾hop completeness for Hotp… view at source ↗
Figure 15
Figure 15. Figure 15: The language control instructions of the Prompt-Control baseline [56, 65]. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The multilingual discourse marks for each language of the DIT baseline. [30, 65]. [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Training time (Llama-3.2-3B & Qwen3-4B back￾bones), Hotpot-ENKB5. Llama3B Qwen4B 70 80 90 ALL-AVG(%) Weak Teacher Strong Teacher [PITH_FULL_IMAGE:figures/full_fig_p037_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: Teacher weight β in TR-RAG, language consis￾tency as metric 0.0002 0.002 0.02 0.2 β 64 66 68 70 char 3-gram r e c a l l ( % ) [PITH_FULL_IMAGE:figures/full_fig_p037_19.png] view at source ↗
Figure 22
Figure 22. Figure 22: Case study. Llama-3.2-3B-Instruct as the backbone, Japanese as language, in Hotpot [PITH_FULL_IMAGE:figures/full_fig_p039_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Case study. Llama-3.2-3B-Instruct as the backbone, Korean as language, in Hotpot [PITH_FULL_IMAGE:figures/full_fig_p039_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Case study. Llama-3.2-3B-Instruct as the backbone, Thai as language, in Hotpot-ENKB5. [PITH_FULL_IMAGE:figures/full_fig_p039_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Case study. Llama-3.2-3B-Instruct as the backbone, Vietnamese as language, in Hotpot [PITH_FULL_IMAGE:figures/full_fig_p039_25.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 1 canonical work pages

  1. [1]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  2. [2]

    XOR QA: Cross-lingual open-retrieval question answering

    Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. XOR QA: Cross-lingual open-retrieval question answering. InProceedings of the 2021 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, pages 547–564, 2021

    2021

  3. [3]

    One question answering model for many languages with cross-lingual dense passage retrieval

    Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. One question answering model for many languages with cross-lingual dense passage retrieval. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information P...

    2021

  4. [4]

    Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    2015

  5. [6]

    Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski

    Jonathan H. Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. Tydi QA: A benchmark for information-seeking question answering in typologically diverse languages.Trans. Assoc. Comput. Linguistics, 8:454–470, 2020

    2020

  6. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  7. [9]

    Cross-lingual self-consistency for multilingual reasoning with language models.arXiv preprint arXiv:2606.01464, 2026

    Ahmed Elhady, Eneko Agirre, and Mikel Artetxe. Cross-lingual self-consistency for multilingual reasoning with language models.arXiv preprint arXiv:2606.01464, 2026

    Pith/arXiv arXiv 2026

  8. [10]

    Retrieval-augmented generation for large language models: A survey.ArXiv, abs/2312.10997, 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.ArXiv, abs/2312.10997, 2023

    Pith/arXiv arXiv 2023

  9. [11]

    Learning word vectors for 157 languages

    Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomáš Mikolov. Learning word vectors for 157 languages. InProceedings of the eleventh international conference on language resources and evaluation (LREC 2018), 2018

    2018

  10. [12]

    MiniLLM: Knowledge distillation of large language models.arXiv preprint arXiv:2306.08543, 2023

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models.arXiv preprint arXiv:2306.08543, 2023. 11

    Pith/arXiv arXiv 2023

  11. [13]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network.CoRR, abs/1503.02531, 2015

    Pith/arXiv arXiv 2015

  12. [14]

    Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin John- son. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. InInternational conference on machine learning, pages 4411–4421. PMLR, 2020

    2020

  13. [15]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP...

    2023

  14. [16]

    doi: 10.18653/V1/2023.EMNLP-MAIN.319

    doi:10.18653/v1/2023.emnlp-main.319 2023

  15. [17]

    RAG-RL: Advancing retrieval-augmented generation via RL and curriculum learning.arXiv preprint arXiv:2503.12759, 2025

    Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, and Tong Zhang. RAG-RL: Advancing retrieval-augmented generation via RL and curriculum learning.arXiv preprint arXiv:2503.12759, 2025

    Pith/arXiv arXiv 2025

  16. [18]

    Learn globally, speak locally: Bridging the gaps in multilingual reasoning.arXiv preprint arXiv:2507.05418, 2025

    Jaedong Hwang, Kumar Tanmay, Seok-Jin Lee, Ayush Agrawal, Hamid Palangi, Kumar Ayush, Ila Fiete, and Paul Pu Liang. Learn globally, speak locally: Bridging the gaps in multilingual reasoning.arXiv preprint arXiv:2507.05418, 2025

    arXiv 2025

  17. [19]

    TinyBERT: Distilling BERT for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. InFindings of the association for computational linguistics: EMNLP 2020, pages 4163–4174, 2020

    2020

  18. [20]

    Bag of tricks for efficient text classification

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL): Volume 2, Short Papers, pages 427–431, 2017

    2017

  19. [21]

    Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating open- domain question answering in the era of large language models. InAnnual Meeting of the Association for Computational Linguistics, 2023

    2023

  20. [22]

    VinePPO: Refining credit assignment in RL training of LLMs.arXiv preprint arXiv:2410.01679, 2024

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs.arXiv preprint arXiv:2410.01679, 2024

    Pith/arXiv arXiv 2024

  21. [23]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016

    2016

  22. [24]

    The power of character n-grams in native language identification

    Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan van Noord, Barbara Plank, and Martijn Wieling. The power of character n-grams in native language identification. In Joel R. Tetreault, Jill Burstein, Claudia Leacock, and Helen Yannakoudakis, editors,Proceed- ings of the 12th Workshop on Innovative Use of NLP for Building Educational Ap...

    work page doi:10.18653/v1/w17-5043 2017

  23. [25]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors,Proceedings of the 29th Symposium on Operating Systems P...

    doi:10.1145/3600006.3613165 2023

  24. [26]

    MLQA: Evaluating cross-lingual extractive question answering

    Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating cross-lingual extractive question answering. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 7315–7330, 2020. 12

    2020

  25. [27]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, edi...

    2020

  26. [28]

    Language drift in multilingual retrieval-augmented gen- eration: Characterization and decoding-time mitigation.arXiv preprint arXiv:2511.09984, 2025

    Bo Li, Zhenghua Xu, and Rui Xie. Language drift in multilingual retrieval-augmented gen- eration: Characterization and decoding-time mitigation.arXiv preprint arXiv:2511.09984, 2025

    arXiv 2025

  27. [30]

    MKQA: A linguistically diverse benchmark for multilingual open domain question answering.Trans

    Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguistically diverse benchmark for multilingual open domain question answering.Trans. Assoc. Comput. Linguistics, 9:1389–1406, 2021

    2021

  28. [31]

    Mmath: A multilingual benchmark for mathematical reasoning.arXiv preprint arXiv:2505.19126, 2025

    Wenyang Luo, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Mmath: A multilingual benchmark for mathematical reasoning.arXiv preprint arXiv:2505.19126, 2025

    Pith/arXiv arXiv 2025

  29. [32]

    When retrieval doesn’t help: A large-scale study of biomedical RAG, 2026

    Erfan Nourbakhsh, Rocky Slavin, Ke Yang, and Anthony Rios. When retrieval doesn’t help: A large-scale study of biomedical RAG, 2026. URLhttps://arxiv.org/abs/2606.04127

    Pith/arXiv arXiv 2026

  30. [33]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  31. [34]

    GRAIL: Gradient-reweighted advantages for reinforcement learning with verifiable rewards, 2026

    Tej Deep Pala, Vernon Toh, and Soujanya Poria. GRAIL: Gradient-reweighted advantages for reinforcement learning with verifiable rewards, 2026. URL https://arxiv.org/abs/2606. 04889

    2026

  32. [35]

    Investigating language preference of multilingual RAG systems.ArXiv, abs/2502.11175, 2025

    Jeonghyun Park and Hwanhee Lee. Investigating language preference of multilingual RAG systems.ArXiv, abs/2502.11175, 2025

    Pith/arXiv arXiv 2025

  33. [36]

    DuDi: Dual-signal distillation with cross-lingual verbalizer,

    Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui, Sarana Nutanong, Alham Fikri Aji, and Peerat Limkonchotiwat. DuDi: Dual-signal distillation with cross-lingual verbalizer,

  34. [37]

    URLhttps://arxiv.org/abs/2606.04694

    Pith/arXiv arXiv

  35. [38]

    chrF: character n-gram F-score for automatic MT evaluation

    Maja Popovi´c. chrF: character n-gram F-score for automatic MT evaluation. In Ond ˇrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors,Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, September 2015. Association for C...

    doi:10.18653/v1/w15-3049 2015

  36. [39]

    chrF++: words helping character n-grams

    Maja Popovi´c. chrF++: words helping character n-grams. In Ond ˇrej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, An- tonio Jimeno Yepes, Philipp Koehn, and Julia Kreutzer, editors,Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark, September 2017. Ass...

    doi:10.18653/v1/w17-4770 2017

  37. [40]

    On the consistency of multilingual context utilization in retrieval-augmented generation.ArXiv, abs/2504.00597, 2025

    Jirui Qi, Raquel Fernández, and Arianna Bisazza. On the consistency of multilingual context utilization in retrieval-augmented generation.ArXiv, abs/2504.00597, 2025

    arXiv 2025

  38. [41]

    Language-coupled reinforcement learning for multilin- gual retrieval-augmented generation.arXiv preprint arXiv:2601.14896, 2026

    Rui Qi, Fengran Mo, Yufeng Chen, Xue Zhang, Shuo Wang, Hongliang Li, Jinan Xu, Meng Jiang, Jian-Yun Nie, and Kaiyu Huang. Language-coupled reinforcement learning for multilin- gual retrieval-augmented generation.arXiv preprint arXiv:2601.14896, 2026

    Pith/arXiv arXiv 2026

  39. [42]

    CroSearch-R1: Better leveraging cross-lingual knowledge for retrieval-augmented generation.arXiv preprint arXiv:2604.25182, 2026

    Rui Qi, Fengran Mo, Sijin Lu, Yufeng Chen, Jian-Yun Nie, and Kaiyu Huang. CroSearch-R1: Better leveraging cross-lingual knowledge for retrieval-augmented generation.arXiv preprint arXiv:2604.25182, 2026. 13

    Pith/arXiv arXiv 2026

  40. [43]

    Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708, 2025

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708, 2025

    Pith/arXiv arXiv 2025

  41. [44]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  42. [46]

    Sequence level training with recurrent neural networks.arXiv preprint arXiv:1511.06732, 2015

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks.arXiv preprint arXiv:1511.06732, 2015

    Pith/arXiv arXiv 2015

  43. [47]

    DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Pith/arXiv arXiv 1910

  44. [48]

    Approximating KL divergence

    John Schulman. Approximating KL divergence. http://joschu.net/blog/kl-approx. html, 2020. Blog post

    2020

  45. [49]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  46. [50]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  47. [51]

    Reinforcement learning-based knowledge distillation with LLM-as-a-judge.arXiv preprint arXiv:2604.02621, 2026

    Yiyang Shen, Lifu Tu, and Weiran Wang. Reinforcement learning-based knowledge distillation with LLM-as-a-judge.arXiv preprint arXiv:2604.02621, 2026

    Pith/arXiv arXiv 2026

  48. [53]

    Patient knowledge distillation for BERT model compression.arXiv preprint arXiv:1908.09355, 2019

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression.arXiv preprint arXiv:1908.09355, 2019

    Pith/arXiv arXiv 1908

  49. [54]

    Policy gradient meth- ods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    1999

  50. [55]

    An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.BMC bioinformatics, 16(1):138, 2015

    George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.BMC bioinformatics, 16(1):138, 2015

    2015

  51. [56]

    MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  52. [57]

    Repro- ducing, analyzing, and detecting reward hacking in rubric-based reinforcement learning, 2026

    Xuekang Wang, Zhuoyuan Hao, Shuo Hou, Hao Peng, Juanzi Li, and Xiaozhi Wang. Repro- ducing, analyzing, and detecting reward hacking in rubric-based reinforcement learning, 2026. URLhttps://arxiv.org/abs/2606.04923

    Pith/arXiv arXiv 2026

  53. [58]

    Polymath: Evaluating mathematical reasoning in multilingual contexts.CoRR, abs/2504.18428, 2025

    Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. Polymath: Evaluating mathematical reasoning in multilingual contexts.CoRR, abs/2504.18428, 2025. doi: 10.48550/ARXIV .2504.18428

    doi:10.48550/arxiv 2025

  54. [59]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 14

    2022

  55. [60]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine learning, 8(3):229–256, 1992

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine learning, 8(3):229–256, 1992

    1992

  56. [61]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  57. [62]

    Self-distillation bridges distribution gap in language model fine-tuning

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1028–1043, Bangkok,...

    2024

  58. [63]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018

    2018

  59. [64]

    Who transfers safety? identifying and targeting cross-lingual shared safety neurons, 2026

    Xianhui Zhang, Chengyu Xie, Linxia Zhu, Yonghui Yang, Weixiang Zhao, Zifeng Cheng, Cong Wang, Fei Shen, and Tat-Seng Chua. Who transfers safety? identifying and targeting cross-lingual shared safety neurons, 2026. URLhttps://arxiv.org/abs/2602.01283

    arXiv 2026

  60. [65]

    MIRACL: A multilingual re- trieval dataset covering 18 diverse languages.Transactions of the Association for Computational Linguistics, 11:1114–1131, 2023

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual re- trieval dataset covering 18 diverse languages.Transactions of the Association for Computational Linguistics, 11:1114–1131, 2023. doi: 10.1162/tacl_a_00595

    doi:10.1162/tacl_a_00595 2023

  61. [66]

    Self-evaluation is already there: Eliciting latent judge calibration in base LLMs with minimal data, 2026

    XiuYu Zhang, Yi Shan, Junfeng Fang, and Zhenkai Liang. Self-evaluation is already there: Eliciting latent judge calibration in base LLMs with minimal data, 2026. URLhttps://arxiv. org/abs/2606.05122

    Pith/arXiv arXiv 2026

  62. [67]

    Think natively: Unlocking multilingual reasoning with consistency- enhanced reinforcement learning.arXiv preprint arXiv:2510.07300, 2025

    Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Kaiyu Huang, Yufeng Chen, Jinan Xu, and Jie Zhou. Think natively: Unlocking multilingual reasoning with consistency- enhanced reinforcement learning.arXiv preprint arXiv:2510.07300, 2025

    arXiv 2025

  63. [68]

    Reinforcement-aware knowledge distillation for LLM reasoning.arXiv preprint arXiv:2602.22495, 2026

    Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware knowledge distillation for LLM reasoning.arXiv preprint arXiv:2602.22495, 2026

    Pith/arXiv arXiv 2026

  64. [69]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026

  65. [70]

    action” is the entire sequencey, and consider asum-KLregularizer (no length normalization). Define a sequence-level teacher “prior

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural In...

    arXiv 2023

  66. [71]

    provides a controlled environment for reproducing and detecting such failures.Unlike prior work that uses these signals only for offline evaluation, we repurpose all three asonline reward componentswithin a unified RL objective, providing complementary training pressure on language adherence, lexical faithfulness, and semantic correctness simultaneously. 42