Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two small language models, at 350M and 1B parameters, out-score every other sub-4B model and match 7–8B rivals on multi-hop RAG benchmarks while grounding answers in literal quote citations.

desk verdict Useful small-model RAG engineering, but the paper's own Annex B disproves its 'systematic grounding' claim. read the letter →

arxiv 2504.18225 v1 pith:TTSIY5BB submitted 2025-04-25 cs.CL

Pierre-Carl Langlais , Pavel Chizhov , Mattia Nee , Carlos Rosas Hinostroza , Matthieu Delsart , Irène Girard , Othman Hicheur , Anastasia Stasenko
show 1 more author
Ivan P. Yamshchikov
This is my paper · ORCID
classification cs.CL
keywords smalllanguagemodelsretrieval-augmentedgenerationcitationmulti-hopquestionansweringsynthetictrainingdatamid-trainingmultilingualinformationretrievalsourcegrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that small language models can become reliable “source reasoners”: mid-trained on a synthetic dataset of roughly 3.1 million retrieval examples (about 9.5 billion tokens), a 350M-parameter and a 1B-parameter model outperform other sub-4B models on the HotpotQA and 2WikiMultiHopQA benchmarks and compete with Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B. The models also generate citations natively, embedding literal quotes wrapped in `` tags during generation, so every factual statement can be checked against the submitted sources. If true, this matters because it moves verifiable, source-grounded question answering onto hardware as small as a Raspberry Pi, and it suggests that citation behavior and retrieval skill can be trained into a model rather than bolted on afterwards. The paper further claims that these are the only small models tested so far that keep their RAG accuracy across French, Italian, German, and Spanish with negligible loss.

What carries the argument

The load-bearing mechanism is the synthetic mid-training pipeline that converts the Common Corpus into an emulated retrieval task. The pipeline has three interlocking parts: back-translation, where a fine-tuned Gemma 3 12B turns a randomly extracted excerpt into a realistic query, issue, or keyword string; emulated retrieval, where BM25 searches a pool of up to 500,000 excerpts so each example contains relevant, partially relevant, and unrelated sources; and structured reasoning traces, generated at scale by a fine-tuned Gemma 12B seeded on 4,000 curated Gemma 27B examples, which force the model through a fixed sequence of analysis, standardized reports, and a draft before the final answer. Adversarial exercises — randomly dropping one to ten sources, shuffling their order, swapping in unrelated queries to train refusals, and translating queries or sources into a mismatched language — are what the authors credit for the models' resilience. The citation behavior itself comes from training the generator to emit literal quotes wrapped in `<ref>` tags during inference, rather than attaching citations after the fact.

What would settle it

Regrade a random sample of the published model outputs on HotpotQA and 2WikiMultiHopQA with human annotators or with an instruction-tuned judge from a different model family, and compare per-model agreement with the reported Gemma 3 12B grades; if the Pleias advantage over Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B narrows or disappears, the benchmark result rests on judge bias rather than model skill. A second check: measure citation precision directly by exact-matching every `<ref>` quote against its stated source, which the reasoning pipeline already makes possible.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a sufficiently well-designed synthetic mid-training run can make sub-billion-parameter models competitive with — and in some cases orthogonal to — models several times their size at retrieval-augmented generation. Pleias-RAG-350M and Pleias-RAG-1B were trained for just under two epochs on roughly 9.5 billion tokens of emulated retrieval: excerpts drawn from the open Common Corpus, back-translated into queries by a fine-tuned Gemma 3 12B, retrieved by BM25 from pools of up to 500,000 excerpts, and augmented with adversarial source shuffling, dropped sources, refusal cases, and cross-lingual translation exercises. The model then follows a fixed reasoning path — query analysis, query report, source analysis, source report, draft — and answers with literal quotes as `<ref>` citations. The paper reports that the 350M model solves roughly 407 HotpotQA questions that both Qwen-2.5-7B and Llama-3.1-8B fail, that both models are Pareto-optimal for RAG accuracy per parameter, and that they are the only tested small models with negligible performance loss on translated HotpotQA in French, Italian, German, and Spanish.

Load-bearing premise

The head-to-head benchmark claims assume that the Gemma 3 12B judge used to grade every model's answers is unbiased across models; because the same model family generated the Pleias training data, a style or format preference in the judge could create or inflate the reported advantage over Qwen, Llama, and Gemma baselines, and no human agreement study is reported.

Editorial extensions

If this is right

  • Sub-billion models become Pareto-optimal for RAG: for a fixed accuracy on multi-hop benchmarks, the Pleias models need an order of magnitude fewer parameters than the next-best open models, making them deployable on edge hardware.
  • Models this small can supplement, not just substitute for, larger models in orchestration: the 350M model solves nearly half of the 864 questions that both Qwen-7B and Llama-8B get wrong.
  • Because citations are generated during inference as literal quotes, downstream systems can audit claims by string-matching each `<ref>` against its source, enabling verification without a second model.
  • English benchmark results should transfer to French, Italian, German, and Spanish deployments, since the models show negligible language-performance loss on translated HotpotQA.
  • The same mid-training recipe — back-translation from an open corpus, adversarial source manipulation, and constrained reasoning traces — should be reproducible by other groups on their own corpora, which the paper explicitly offers as an open methodology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation design leaves a judge-bias channel open: the same model family (Gemma 3) that generated the synthetic training data also grades the answers. A future replication that re-judges the published generations with human raters or a non-Gemma judge, or that checks whether the judge rewards citation formatting itself, would settle whether the headline advantage is real; the paper reports no
  • The headline results are all “trivial mode” queries with short answers, as the paper itself notes. Long-form synthesis and deep-research-style tasks — where citation grounding would be most valuable — remain untested, so a long-form RAG benchmark with exact-quote verification would be a discriminating next experiment.
  • The paper itself acknowledges a persistent failure mode in which the model drifts into answering a related question when the exact answer is absent from the sources; this weakens the refusal guarantee in production and is a natural target for the planned reinforcement-learning stage.
  • If the recipe transfers, the most consequential effect is a change in who can build grounded QA systems: organizations with sensitive or proprietary corpora could mid-train small open models on their own documents and deploy them on-device without sending data to closed APIs, extending the deployment pattern the paper demonstrates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Pleias-RAG-350m and Pleias-RAG-1B, two small language models obtained by mid-training Pleias 1.0 base models on a synthetic dataset of roughly 3.1 million RAG examples (about 9.5 billion tokens) built from the Common Corpus. The models generate answers in a structured reasoning format with native <ref>-tagged citations, and the authors claim state-of-the-art performance among sub-4B models on HotpotQA and 2WikiMultiHopQA, competitiveness with larger models such as Qwen-2.5-7B and Llama-3.1-8B, negligible multilingual degradation on translated HotpotQA in four European languages, and systematic reference grounding for statements. The evaluation relies on Gemma 3 12B as an LLM-as-a-match judge, with results presented through figures rather than numeric tables. The paper also includes two annexes with worked examples, one of which (Annex B) is offered as evidence of cross-lingual reasoning and grounding.

Significance. If the headline claims were established, the work would be practically significant: it would demonstrate that sub-1B models can perform competitive multi-hop RAG with verifiable citations on constrained hardware, and the authors explicitly release the models and the evaluation set. The training-data transparency (Common Corpus, permissible licenses) and the focus on source-grounded generation are also strengths. However, the central claim of systematic reference grounding is directly contradicted by the paper's own curated success case in Annex B, where the model's cited answer asserts something the quoted sources explicitly deny. The benchmark claims are additionally supported only by figures without numeric values, confidence intervals, significance tests, or a human agreement study for the LLM judge. These problems affect the core contributions, so the significance of the work as presented is not established.

major comments (4)
  1. [Annex B] Annex B, presented as evidence of successful cross-lingual reasoning, directly contradicts the abstract's claim of 'systematic reference grounding for statements.' The model's answer states that Lennon–McCartney is the original artist who recorded 'Act Naturally,' and its query analysis repeats this misattribution. The cited source 6 says only that 'If You've Got Trouble' was written by Lennon–McCartney and that the Beatles chose 'Act Naturally' instead; source 10, which the model also cites, explicitly says 'Act Naturally' was written by Johnny Russell and Voni Morrison and originally recorded by Buck Owens and the Buckaroos. The cited evidence therefore contradicts the answer, and the answer is factually incorrect. Because this is the paper's own curated success case, the evidence offered in the manuscript itself is inconsistent with the central grounding claim, independent of any benchmark methodology debate.
  2. [Section 4.1] The evaluation uses an LLM-as-a-match judge, Gemma 3 12B, that belongs to the same model family that generated the synthetic training data (Section 3.2 and Section 3.4). The paper reports no human agreement study, no per-model judge bias analysis, and no ablation showing that the judge's grades correlate with human correctness judgments. In addition, the acceptance threshold is not uniform: 'yes' only for HotpotQA and 2WikiMultiHopQA, but 'yes' and 'rather yes' for MuSiQue. Given these issues, the claimed outperformance over Qwen, Llama, and Gemma baselines may be attributable to judge bias in favor of outputs that resemble the synthetic generations, and the benchmark claims in the abstract are not established.
  3. [Section 4.2 and Figures 8-9] The central quantitative claims are presented only through figures, with no numeric scores, no sample sizes for the reported percentages, no confidence intervals, and no significance tests. For example, the statement that Pleias models are 'currently SOTA on 2WikiMultiHopQA' and 'occupy the Pareto-optimal zone' cannot be verified or compared against future work without the underlying numbers. The paper also does not report the decoding settings, prompt templates, or number of runs used for the baselines, which are necessary to interpret comparisons across models of different sizes and formats.
  4. [Section 3.2 and Section 4.1] The authors state that Wikipedia is a 'neutralized source' because it is universally used as training data, but the mid-training set includes 'contemporary web corpora (especially Wikipedia)' drawn from Common Corpus, while the benchmark questions are derived from Wikipedia (HotpotQA, 2WikiMultiHopQA, MuSiQue). No overlap analysis is provided to show that the specific benchmark passages or paraphrases were not present in the mid-training data. Because the advantage of the Pleias models is partially attributed to retrieval from provided sources, the possibility of memorization-based shortcuts should be addressed quantitatively; otherwise the 'neutralized source' claim is an unsupported assumption.
minor comments (5)
  1. [Section 4.2] The name 'Pleias-RAG-1.2B' is used in Section 4.2, while the title and abstract consistently say 'Pleias-RAG-1B'; please standardize the model naming.
  2. [Section 4.2] Figure 9's caption text includes 'language conversation performance,' which appears to be a typo for 'language conversion performance' or 'language conservation performance.'
  3. [Section 6] The phrase 'common pitfall (likelost in the middle' has a formatting error and should read 'common pitfall, like lost in the middle.'
  4. [Section 5.2] The deployment description 'Raspberry Pi 4 (8 giga ram)' should be written as 'Raspberry Pi 4 with 8 GB RAM' for clarity.
  5. [Section 4.2] The claim that all benchmark queries correspond to the model's 'trivial' mode is important context for the reported results and should be stated earlier, ideally in the evaluation setup, so that readers can calibrate the difficulty of the task.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: benchmark claims rest on independent datasets, and the Gemma-in-the-loop and Annex B issues are validity/correctness concerns, not circular steps.

full rationale

The paper makes no derivational claim that reduces to its own inputs. The headline benchmark comparisons are run against external, pre-existing datasets (HotpotQA, 2WikiMultiHopQA, MuSiQue), with the model's training-data generation and answer judging both involving Gemma 3. That overlap is a real risk of judge bias, but it is not a construction-level equivalence: no benchmark parameter is fitted from the judge's outputs, and the questions and ground-truth answers come from independent benchmarks. The self-citations (Pleias 1.0 base models, Chizhov et al. 2025, Langlais 2025a/b) are contextual and not load-bearing; no uniqueness theorem or prior result is invoked to forbid alternatives. The internal inconsistency in Annex B, where a cited answer attributes 'Act Naturally' to Lennon–McCartney while the quoted source 10 says Buck Owens originally recorded it, undermines the 'systematic reference grounding' claim empirically, but it is a correctness failure rather than a circular derivation. Accordingly, no circular step meets the required evidentiary bar of exhibiting a specific reduction of a prediction to its fitted or self-cited inputs, so the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central performance claims rest on the validity of the Gemma judge, the licensing and quality of Common Corpus and synthetic data, the absence of benchmark contamination, and the stability of base-model skills after mid-training. The hand-chosen thresholds listed above directly affect the reported benchmark outcomes; they are not derived from first principles.

free parameters (5)
  • MuSiQue acceptance threshold = 'yes' or 'rather yes' for MuSiQue; 'yes' only for HotpotQA and 2WikiMultiHopQA
    The paper widens accepted judge grades for the hardest benchmark after seeing its difficulty. This hand-chosen rule directly affects the reported competitive position on MuSiQue.
  • Refusal swap proportion = 5% of the retrieval dataset
    Randomly unrelated queries are swapped in to train refusals; the proportion is chosen by hand without ablations, and it shapes the refusal behavior that is part of the qualitative claims.
  • Language switching subsample sizes = 5% translated queries + 5% translated sources = 10%
    Two randomized subsamples are used for the translation exercise; this hand-set share plausibly drives the multilingual robustness claim and is not ablated.
  • Mid-training duration = slightly less than 2 epochs (~9.47B tokens)
    The stopping point is chosen by loss behavior, not by a validated criterion; loss was still decreasing at the end, so the reported model may be undertrained or overtrained relative to an optimum.
  • Learning rate = about one order of magnitude higher than the final pretraining learning rate
    Selected after tests; no sweep or sensitivity analysis is reported, and this affects the benchmark results.
assumptions (5)
  • domain assumption The Common Corpus and Gemma synthetic outputs are licensed such that training and release are lawful.
    The paper asserts open-license reuse repeatedly (Sections 1, 3, 5.4) but does not provide license documentation or legal verification for every corpus subset and synthetic output.
  • domain assumption Gemma 3 12B instruction-tuned output is a valid answer-correctness judge for these benchmarks, with no human agreement required.
    Section 4.1 grades every submission with this judge and reports no inter-annotator or judge-human agreement.
  • domain assumption Literal quote inclusion is sufficient to verify grounding; claim-level entailment between answer and quote is not separately checked.
    Section 3.4 filters hallucinated citations and re-extracts quotes, but Annex B shows a quote contradicting the answer, so quote presence alone does not guarantee grounding.
  • domain assumption Training on Common Corpus (which includes Wikipedia) does not contaminate HotpotQA, 2wiki, or MuSiQue evaluations.
    Section 4.1 calls Wikipedia a neutralized source but provides no contamination analysis.
  • domain assumption Mid-training on the full 10B-token set for almost two epochs preserves base model skills and does not cause harmful forgetting.
    Section 3.5 reports loss still decreasing at the end of training; no downstream task stability or forgetting checks are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family." pith.science (2026). https://pith.science/paper/TTSIY5BB

@misc{pith2026250418225,
  author       = {Pith},
  title        = {Pith review of: Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTSIY5BB}},
  note         = {Machine review of arXiv:2504.18225}
}
read the original abstract

We introduce a new generation of small reasoning models for RAG, search, and source summarization. Pleias-RAG-350m and Pleias-RAG-1B are mid-trained on a large synthetic dataset emulating the retrieval of a wide variety of multilingual open sources from the Common Corpus. They provide native support for citation and grounding with literal quotes and reintegrate multiple features associated with RAG workflows, such as query routing, query reformulation, and source reranking. Pleias-RAG-350m and Pleias-RAG-1B outperform SLMs below 4 billion parameters on standardized RAG benchmarks (HotPotQA, 2wiki) and are competitive with popular larger models, including Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B. They are the only SLMs to date maintaining consistent RAG performance across leading European languages and ensuring systematic reference grounding for statements. Due to their size and ease of deployment on constrained infrastructure and higher factuality by design, the models unlock a range of new use cases for generative AI.

Figures

Figures reproduced from arXiv: 2504.18225 by the authors.

Figure 1
Figure 1. Scores on HotPotQA evaluation versus model size. Both Pleias models are Pareto-optimal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of a reconstruction of Anthropic citation mode [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Main scenarios incorporated into the reasoning model: trivial question (with a shortened [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Standardized RAG workflow integrated within the model also featuring further options [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Token reassignment strategy for the RAG specialized models. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Simplified workflow of our retrieval strategy. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Training run of Pleias-RAG-350M. 4 Evaluation 4.1 Standard benchmark We use three established benchmarks to assess advanced information retrieval: HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022). The three benchmarks …
Figure 8
Figure 8. Figure 8: Results of standard evaluation on English benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Estimate of language performance loss in four European languages (French, Spanish, [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 13 canonical work pages

  1. [1]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

    arXiv 2024

  2. [2]

    Building Effective AI Agents

    Anthropic. Building Effective AI Agents . URL https://www.anthropic.com/engineering/building-effective-agents

    work page

  3. [3]

    Circuit Tracing : Revealing Computational Graphs in Language Models

    Anthropic. Circuit Tracing : Revealing Computational Graphs in Language Models . In Transformer Circuits . Anthropic, 2025 a . URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html

    work page 2025

  4. [4]

    Introducing Citations on the Anthropic API , 2025 b

    Anthropic. Introducing Citations on the Anthropic API , 2025 b . URL https://www.anthropic.com/news/introducing-citations-api

    work page 2025

  5. [5]

    Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. ReSearch : Learning to Reason with Search for LLMs via Reinforcement Learning , March 2025. URL http://arxiv.org/abs/2503.19470. arXiv:2503.19470 [cs]

    arXiv 2025

  6. [6]

    Yamshchikov

    Pavel Chizhov, Mattia Nee, Pierre-Carl Langlais, and Ivan P. Yamshchikov. What the HellaSwag ? On the Validity of Common - Sense Reasoning Benchmarks , April 2025. URL http://arxiv.org/abs/2504.07825. arXiv:2504.07825 [cs]

    arXiv 2025

  7. [7]

    Orchestrating Synthetic Data with Reasoning

    Tim R Davidson, Benoit Seguin Enrico Bacis, Cesar Ilharco, and Hamza Harkous. Orchestrating Synthetic Data with Reasoning . In ICLR 2025 Workshop on Synth Data, 2025

    work page 2025

  8. [8]

    DeepSeek - R1 : Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , January 2025

    DeepSeek-AI. DeepSeek - R1 : Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , January 2025. URL http://arxiv.org/abs/2501.12948. arXiv:2501.12948 [cs]

    arXiv 2025

Show all 40 references
  1. [9]

    Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D. Manning. Synthetic Data Generation & Multi - Step RL for Reasoning & Tool Use , April 2025. URL http://arxiv.org/abs/2504.04736. arXiv:2504.04736 [cs]

    2025 arXiv

  2. [10]

    Surveying the Effects of Quality , Diversity , and Complexity in Synthetic Data From Large Language Models , December 2024

    Alex Havrilla, Andrew Dai, Laura O'Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fabrizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, Duy Phung, Maia Iyer, Dakota Mahan, Chase Blagden, Srishti Gureja, Mohammed Hamdy, Wen-Ding Li, Giovanni Paolini, Paw...

    2024 arXiv

  3. [11]

    Constructing A Multi -hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , November 2020

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A Multi -hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , November 2020. URL http://arxiv.org/abs/2011.01060. arXiv:2011.01060 [cs] version: 2

    2020 arXiv

  4. [12]

    Large language models cannot self-correct reasoning yet, 2024

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2024. URL https://arxiv.org/abs/2310.01798

    2024 arXiv

  5. [13]

    Team et al. Kimi. Kimi k1.5: Scaling Reinforcement Learning with LLMs , March 2025. URL http://arxiv.org/abs/2501.12599. arXiv:2501.12599 [cs]

    2025 arXiv

  6. [14]

    The Model is the Product , 2025 a

    Pierre-Carl Langlais. The Model is the Product , 2025 a . URL https://vintagedata.org/blog/posts/model-is-the-product

    2025

  7. [15]

    What's the deal with mid-training?, 2025 b

    Pierre-Carl Langlais. What's the deal with mid-training?, 2025 b . URL https://vintagedata.org/blog/posts/model-is-the-product

    2025

  8. [16]

    Improving Attributed Text Generation of Large Language Models via Preference Learning , March 2024

    Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, and Min Zhang. Improving Attributed Text Generation of Large Language Models via Preference Learning , March 2024. URL http://arxiv.org/abs/2403.18381. arXiv:2403.18381 [cs]

    2024 arXiv

  9. [17]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best Practices and Lessons Learned on Synthetic Data , August 2024. URL http://arxiv.org/abs/2404.07503. arXiv:2404.07503 [cs]

    2024 arXiv

  10. [18]

    Inference-time scaling for generalist reward modeling, 2025

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling, 2025. URL https://arxiv.org/abs/2504.02495

    2025

  11. [19]

    Scaling Laws for Fact Memorization of Large Language Models , June 2024

    Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling Laws for Fact Memorization of Large Language Models , June 2024. URL https://arxiv.org/abs/2406.15720v1

    2024 arXiv

  12. [20]

    Varma, Hang Zou, Qiyang Zhao, and Merouane Debbah

    Thomas Mongaillard, Samson Lasaulce, Othman Hicheur, Chao Zhang, Lina Bariah, Vineeth S. Varma, Hang Zou, Qiyang Zhao, and Merouane Debbah. Large language models for power scheduling: A user-centric approach, 2024. URL https://arxiv.org/abs/2407.00476

    2024 arXiv

  13. [21]

    Synthetic- Based Retrieval of Patient Medical

    Rinat Mullahmetov and Ilya Pershin. Synthetic- Based Retrieval of Patient Medical . In ICLR 2025 Workshop on Synth Data, 2025

    2025

  14. [22]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2025 arXiv

  15. [23]

    Introducing deep research OpenAI

    OpenAI. Introducing deep research OpenAI . Technical report, OpenAI, 2025. URL https://openai.com/index/introducing-deep-research/

    2025

  16. [24]

    On the Capacity of Citation Generation by Large Language Models , October 2024

    Haosheng Qian, Yixing Fan, Ruqing Zhang, and Jiafeng Guo. On the Capacity of Citation Generation by Large Language Models , October 2024. URL http://arxiv.org/abs/2410.11217. arXiv:2410.11217 [cs]

    2024 arXiv

  17. [25]

    Qwen2.5 Technical Report , January 2025

    Qwen. Qwen2.5 Technical Report , January 2025. URL http://arxiv.org/abs/2412.15115. arXiv:2412.15115 [cs]

    2025 arXiv

  18. [26]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners

  19. [27]

    TraceBack -12b, March 2025

    Secemp. TraceBack -12b, March 2025. URL https://huggingface.co/secemp9/TraceBack-12b

    2025

  20. [28]

    Improving Neural Machine Translation Models with Monolingual Data

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving Neural Machine Translation Models with Monolingual Data . In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , p...

    2016 doi

  21. [29]

    Citekit: A Modular Toolkit for Large Language Model Citation Generation , December 2024

    Jiajun Shen, Tong Zhou, Yubo Chen, and Kang Liu. Citekit: A Modular Toolkit for Large Language Model Citation Generation , December 2024. URL http://arxiv.org/abs/2408.04662. arXiv:2408.04662 [cs]

    2024 arXiv

  22. [30]

    Gemma 3 Technical Report , March 2025

    Gemma Team. Gemma 3 Technical Report , March 2025. URL http://arxiv.org/abs/2503.19786. arXiv:2503.19786 [cs]

    2025 arXiv

  23. [31]

    Hugo et al. Touvron. The Llama 3 Herd of Models , November 2024. URL http://arxiv.org/abs/2407.21783. arXiv:2407.21783 [cs]

    2024 arXiv

  24. [32]

    Towards Internet - Scale Training For Agents , February 2025

    Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards Internet - Scale Training For Agents , February 2025. URL http://arxiv.org/abs/2502.06776. arXiv:2502.06776 [cs]

    2025 arXiv

  25. [33]

    MuSiQue : Multihop Questions via Single -hop Question Composition , May 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue : Multihop Questions via Single -hop Question Composition , May 2022. URL http://arxiv.org/abs/2108.00573. arXiv:2108.00573 [cs]

    2022 arXiv

  26. [34]

    A Short Survey on Small Reasoning Models : Training , Inference , Applications and Research Directions , April 2025

    Chengyu Wang, Taolin Zhang, Richang Hong, and Jun Huang. A Short Survey on Small Reasoning Models : Training , Inference , Applications and Research Directions , April 2025. URL https://arxiv.org/abs/2504.09100v1

    2025 arXiv

  27. [35]

    A Comprehensive Survey of Small Language Models in the Era of Large Language Models - Techniques , Enhancements , Applications , Collaboration with LLMs , and Trustworthiness

    Fali Wang. A Comprehensive Survey of Small Language Models in the Era of Large Language Models - Techniques , Enhancements , Applications , Collaboration with LLMs , and Trustworthiness . 2025

    2025

  28. [36]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. A Comprehensive Survey of Small Language Models in the Era of Large Language Models : Techniques , Enhancements ...

    2024 arXiv

  29. [37]

    xjdr-alt/entropix, April 2025

    doomslide xjdr. xjdr-alt/entropix, April 2025. URL https://github.com/xjdr-alt/entropix. original-date: 2024-10-03T01:02:51Z

    2025

  30. [38]

    Stronger Models Are Not Always Stronger Teachers for Instruction Tuning

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Stronger Models Are Not Always Stronger Teachers for Instruction Tuning . In ICLR 2025 Workshop on Synth Data, 2025

    2025

  31. [39]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA : A Dataset for Diverse , Explainable Multi -hop Question Answering , September 2018. URL http://arxiv.org/abs/1809.09600. arXiv:1809.09600 [cs]

    2018 arXiv

  32. [40]

    Effective Large Language Model Adaptation for Improved Grounding and Citation Generation , April 2024

    Xi Ye, Ruoxi Sun, Sercan Ö Arik, and Tomas Pfister. Effective Large Language Model Adaptation for Improved Grounding and Citation Generation , April 2024. URL http://arxiv.org/abs/2311.09533. arXiv:2311.09533 [cs]

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.