Pith. sign in

REVIEW 4 major objections 4 minor 56 references

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A retrieval-augmented, chain-of-thought preference dataset aligned with DPO makes a small language model more knowledgeable and more transparent in Traditional Chinese Medicine, beating both the base model and SFT-only training.

desk verdict The pipeline is sensible and the write-up is honest, but the central accuracy claim is contradicted by the paper's own tables. read the letter →

arxiv 2506.02726 v1 pith:W6JKV3MV submitted 2025-06-03 cs.CL cs.AIcs.LG

Qihang Yan , Xinyu Zhang , Luming Guo , Qi Zhang , Feifan Liu This is my paper
classification cs.CLcs.AIcs.LG
keywords LargeLanguageModelsPreferenceAlignmentDirectOptimizationRetrieval-AugmentedGenerationChain-of-ThoughtModelInterpretabilityVerticalDomainsTraditionalChineseMedicine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RACE-Align is a recipe for preference data that teaches a language model not just which answer is better, but which way of reasoning is better. For each question it constructs a chosen response from retrieved external knowledge plus an explicit chain-of-thought, and a rejected response produced by a second model answering directly, then aligns the target model with Direct Preference Optimization over those pairs. Tested on 5,000 Traditional Chinese Medicine QA entries with a 1.7-billion-parameter base model, the aligned model scores higher than both the base model and an SFT-only model on information richness, relevance, reasoning logicality and depth, and interpretability in blind expert evaluation. The paper's point is that grounding preferred answers in retrieved facts and treating the reasoning trace itself as a preference dimension is a cost-effective route to domain competence and process transparency.

What carries the argument

The load-bearing object is the preference dataset $D = \{(q_i, y_i^w, y_i^l)\}$ in which every response is a pair (reasoning, answer) and the chosen response is produced in two sub-steps: first a chain-of-thought is written from the enhanced question plus retrieved knowledge, then the final answer is written to follow that chain. The rejected response is a direct LLM answer together with its raw thought process, and both responses are wrapped in thinking tags when formatted for training. The DPO objective $L_{\mathrm{DPO}}(\pi_\theta; \pi_{\mathrm{ref}})$ then pushes the model to raise the likelihood ratio of the chosen (reasoning, answer) pair over the rejected pair, so the reasoning trace itself is trained on rather than only the final output.

What would settle it

Have TCM experts blindly compare the chosen and rejected responses from the constructed dataset without knowing which is which; if the chosen responses are not preferred at a clearly above-chance rate, the preference signal RACE-Align optimizes is not real. A second check is to train DPO with the labels swapped and show the model performs equally well, which would indicate the improvement comes from the data content rather than the preference direction.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quality of an LLM's reasoning process can be optimized directly by making it part of the preference signal. RACE-Align builds a dataset of triples (prompt, chosen, rejected) in which both the chosen and rejected responses contain an explicit reasoning trace: the chosen trace is written from the enhanced question plus retrieved external knowledge, the chosen answer then follows that trace, and the rejected response is a direct, less grounded generation with its own raw thought process. Applying the standard DPO loss to this dataset increases the likelihood ratio of the retrieval-grounded, chain-of-thought-structured response over the direct one. In the TCM experiments the resulting model beats the base model and the SFT model on most human-evaluated dimensions, including TCM thinking patterns, reasoning logicality and depth, information richness, and interpretability, while the base model keeps the top raw factuality score.

Load-bearing premise

The method rests on the assumption that the answers built from outside information and step-by-step reasoning really are better than the direct answers marked as rejected—an assumption the paper states but never checks with independent human judges on the raw pairs.

Editorial extensions

If this is right

  • If RACE-Align works as claimed, a domain LLM can be improved without human preference annotation, because the chosen and rejected pairs are produced by an AI-driven pipeline combining question rewriting, web search, and instruction-following generation.
  • The aligned model's advantage in TCM thinking patterns and reasoning depth indicates that putting explicit chain-of-thought in the training signal shapes the model's displayed reasoning, not just the wording of its answers.
  • Because the DPO reference policy is the SFT model, RACE-Align functions as an alignment layer on top of supervised fine-tuning rather than a replacement for it.
  • The multi-stage pipeline is presented as transferable to other vertical domains such as law or finance, where retrieved knowledge and professional reasoning patterns are equally important.
  • Both automated metrics (ROUGE-L, BLEU-4, and LLM-based scores) and blind human ratings favor the RACE-Align model, so the claimed improvement is not an artifact of a single evaluation mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run the ablation that would isolate the reasoning-trace component: DPO on the same chosen and rejected final answers without their reasoning tags. If the gap over SFT collapsed, the explicit chain-of-thought preference would be what carries the gain; if it held, the gain could come from the retrieval-grounded answer content alone.
  • The entire method depends on the AI-generated chosen answers being genuinely better than the rejected ones, yet the five TCM experts rated complete model outputs, not the raw preference pairs; a small blind pairwise test on the pairs themselves would settle that dependency.
  • Since the base model kept the highest raw factuality score, a natural refinement is to combine RACE-Align with a factuality-preserving term or a mixed preference signal; the paper leaves that combination untested.
  • If reasoning-trace preference generalizes, the same data recipe could apply wherever process transparency is the product, such as clinical reasoning or legal analysis, but cross-domain transfer is not measured in this paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RACE-Align, a framework for constructing binary preference data that combines retrieval-augmented generation (RAG) and explicit chain-of-thought (CoT) reasoning, and then aligning a base LLM with Direct Preference Optimization (DPO). The framework is applied to a Traditional Chinese Medicine (TCM) question-answering setting using Qwen3-1.7B as the base model, with Gemini 2.5 Flash generating the preferred answers and Qwen3-235B-A22B generating the rejected answers. The authors compare the RACE-Align DPO model against the base model and an SFT-only model using both automated metrics (Gemini 2.5 Pro scoring, ROUGE, BLEU) and human evaluation by five TCM-background volunteers on 100 validation samples. The paper claims significant improvements in answer accuracy, information richness, TCM thinking pattern application, reasoning logicality and depth, and interpretability.

Significance. If the claimed improvements were supported by the data, RACE-Align would offer a practical, cost-effective recipe for aligning small LLMs in vertical domains by encoding external knowledge and reasoning process preferences into DPO training. The data-generation pipeline is described in detail and is largely a reasonable composition of existing techniques (RAG, CoT, DPO), though it does not release the constructed dataset or code, and the evaluation is too weak to establish the headline claims. The central empirical claim is directly contradicted by the paper's own tables on the accuracy/factuality dimension, and the statistical basis for 'significantly outperforms' is absent. As such, the contribution, if any, is not convincingly demonstrated in the current manuscript.

major comments (4)
  1. [Abstract and Tables 1–2] The abstract states that RACE-Align 'significantly outperforms the original base model and a model fine-tuned only with Supervised Fine-Tuning (SFT)' with improvements 'including answer accuracy.' This is contradicted by the paper's own results: Table 1 reports automated accuracy of 8.0 for RACE-Align DPO versus 8.5 for the base model, and Table 2 reports human 'Answer Accuracy & Factuality' of 3.8 versus 4.2 for the base model. On the accuracy dimension, which is the most directly tied to factual grounding, RACE-Align is numerically below the base model. The conclusion (§7) similarly claims superiority in 'answer knowledgeability,' which is not supported by the factuality scores. This is a load-bearing inconsistency that must be resolved before any claim of broad superiority can be accepted.
  2. [§4.4 and §5.2] The claim of 'significant' improvement over the base model and the SFT model is not backed by any statistical testing. The human evaluation uses 5 raters on 100 samples and reports only mean Likert scores without confidence intervals, paired significance tests (e.g., Wilcoxon signed-rank test), or inter-rater reliability (e.g., Krippendorff's alpha or Cohen's kappa). With this evaluation design, differences of 0.2–0.3 points on a 5-point scale are not interpretable as significant. The paper itself acknowledges in the Discussion (§6) that the number of evaluators was 'restricted,' but no attempt is made to quantify the uncertainty. Consequently, the repeated use of 'significantly outperforms' (Abstract, §7) is unsubstantiated.
  3. [§4.4 and §3.2] The automated evaluation is potentially circular: Gemini 2.5 Pro is used as the automated judge, while the preferred answers in the training data were generated by Gemini 2.5 Flash (with RAG and CoT). The human evaluators also see the full responses, including the long retrieval-augmented and CoT-structured answers that the chosen model produces. This confound makes it difficult to attribute the higher scores on information richness, logicality, and interpretability to genuine reasoning quality rather than to format, length, or evaluator bias toward the more elaborate structure. The paper does not control for answer length or format, nor does it independently validate that the 'preferred' responses are actually better than the 'rejected' ones using human judges. Without such a check, the DPO preference signal itself rests on an unverified assumption.
  4. [§3.1] The entire method relies on the assumption that the AI-generated preferred answers (retrieval-grounded, CoT-structured) are higher quality than the rejected answers (direct Qwen responses). The paper states this as a design assumption but never validates it, e.g., by having human experts compare a sample of chosen and rejected pairs without knowing their source. If the assumption fails, the preference data are meaningless and the DPO training (Eq. 9) optimizes an invalid likelihood ratio. Given that the chosen answers are systematically longer and retrieved, and that the human accuracy score for the final aligned model is below the base model's, this assumption is not self-evidently satisfied and requires explicit empirical verification.
minor comments (4)
  1. [Title and throughout] The title on the first page reads 'Chain-of-Thoughten' (missing a space) and many words are concatenated, e.g., 'LARGELANGUAGEMODELS.' These formatting errors should be corrected.
  2. [§5.1] The sentence 'The SFT Model's performance was intermediate across various metrics but was lower in accuracy than the RACE-Align DPO model' is misleading because the base model has the highest accuracy; the text should clarify that RACE-Align accuracy is between SFT and base.
  3. [§4.1] The validation set is the last 100 pairs from the same data-generation pipeline as the training set; this does not constitute a fully independent out-of-domain evaluation, and the paper should discuss potential distribution overlap.
  4. [References] Some references appear incomplete or informal (e.g., [45] has no publication venue, [26] cites OpenReview without an ID, [54] cites a GitHub repository without version or access date). These should be completed for reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

Validation-set hyperparameter tuning makes the reported 'unseen data' evaluation a fitted result; central comparison is not independent.

  1. fitted input called prediction [Section 4.1, 4.3, 4.4 (Dataset Construction, Training Details, Evaluation Methods)]
    "From the finally generated 5000 high-quality preference data pairs, we selected the first 4900 as the training set and the latter 100 as the validation set, used to evaluate the model’s generalization ability on unseen data. ... Hyperparameters such as learning rate, batch size, and training epochs were selected through preliminary experiments on the validation set."

    The same 100 'validation' examples are used twice: first to select hyperparameters (learning rate, batch size, epochs, and DPO β), then as the evaluation set for Tables 1-2 and the human scores. A configuration tuned on these examples cannot be said to generalize to unseen data; the reported scores are fitted to the evaluation subset. Because the abstract and conclusion claim RACE-Align 'significantly outperforms' on the basis of these scores, the central empirical claim reduces to a model-selection artifact rather than an independent test.

full rationale

The paper's only formal circularity is in the evaluation protocol. Section 4.1 calls the last 100 preference pairs the validation set 'used to evaluate the model’s generalization ability on unseen data,' but Section 4.3 says hyperparameters—including learning rate, batch size, epochs, and DPO β—were selected by experiments on exactly that validation set, and Section 4.4 reports the final comparison on those same 100 samples. Thus Tables 1 and 2 are not a test on unseen data; the configuration was fitted to the evaluation sample, so the claim that RACE-Align 'significantly outperforms' is partly forced by model selection. This is a real but bounded circularity: it biases the reported numbers rather than making the DPO update itself tautological. Additional concerns—Gemini being used both to generate preferred answers and as automated judge, and preference labels being assumed rather than human-verified—are independence and validity risks, not by-construction equivalences, and are not counted as separate circular steps. Separately, the abstract's 'answer accuracy' claim is contradicted by the paper's own Tables 1 and 2, but that is a correctness issue, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the quality of the AI-generated preference data and on the evaluation protocol. No new theoretical entities are introduced. The free parameters are standard training hyperparameters, but the lack of reported values limits reproducibility.

free parameters (3)
  • DPO beta (β)
    Controls the KL regularization strength in the DPO loss (Equation 9). Adjusted based on literature and preliminary experiments, but value not given (Section 4.3).
  • SFT/DPO training hyperparameters (learning rate, batch size, epochs)
    Selected via preliminary experiments on the validation set; exact values omitted (Section 4.3).
  • Train/validation split = 4900/100
    First 4900 samples for training, remainder for validation; arbitrary split (Section 4.1).
assumptions (3)
  • domain assumption High-quality answers in vertical domains must conform to domain-specific reasoning logic
    Stated as a key design assumption in Section 3.1; not empirically justified.
  • ad hoc to paper AI-generated preferred answers (Gemini with RAG+CoT) are superior to rejected answers (Qwen without RAG)
    The entire method depends on this preference label quality, but no human validation of the preference pairs is reported (Section 3.2).
  • domain assumption Gemini 2.5 Pro provides reliable automated evaluation scores
    Used as the automated judge (Section 4.4) with no evidence of agreement with human judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models." pith.science (2026). https://pith.science/paper/W6JKV3MV

@misc{pith2026250602726,
  author       = {Pith},
  title        = {Pith review of: RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6JKV3MV}},
  note         = {Machine review of arXiv:2506.02726}
}
read the original abstract

Large Language Models (LLMs) struggle with accuracy, domain-specific reasoning, and interpretability in vertical domains. Traditional preference alignment methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) often overlook the underlying knowledge sources and reasoning logic. This paper introduces RACE-Align (Retrieval-Augmented and Chain-of-Thought Enhanced Alignment), a novel framework designed to address these limitations. RACE-Align systematically constructs a binary preference dataset incorporating external knowledge support and explicit Chain-of-Thought (CoT) reasoning, then aligns LLMs using the DPO algorithm. The core innovation lies in its preference data construction strategy: it integrates AI-driven retrieval for factual grounding, enhancing knowledgeability and accuracy, and emphasizes the optimization of domain-specific CoT, treating the reasoning process itself as a key preference dimension. A multi-stage, AI-driven refinement pipeline cost-effectively generates these preference pairs. Experimental validation in Traditional Chinese Medicine (TCM) using Qwen3-1.7B as the base model demonstrates that RACE-Align significantly outperforms the original base model and a model fine-tuned only with Supervised Fine-Tuning (SFT). Improvements were observed across multiple dimensions, including answer accuracy, information richness, application of TCM thinking patterns, logicality and depth of reasoning, and interpretability. These findings suggest RACE-Align offers an effective pathway to enhance LLMs' knowledge application, reasoning reliability, and process transparency in complex vertical domains.

Figures

Figures reproduced from arXiv: 2506.02726 by the authors.

Figure 1
Figure 1. The RACE-Align framework flowchart. The process begins with an original user question, which is [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 22 canonical work pages

  1. [1]

    Large language models: A survey of their development, capabilities, and applications.Knowledge and Information Systems, pages 1–56, 2024

    Yadagiri Annepaka and Partha Pakray. Large language models: A survey of their development, capabilities, and applications.Knowledge and Information Systems, pages 1–56, 2024

    work page 2024

  2. [2]

    Nlp for social good: A survey of challenges, opportunities, and responsible deployment.arXiv preprint arXiv:2505.22327, 2025

    Antonia Karamolegkou, Angana Borah, Eunjung Cho, Sagnik Ray Choudhury, Martina Galletti, Rajarshi Ghosh, Pranav Gupta, Oana Ignat, Priyanka Kargupta, Neema Kotonya, et al. Nlp for social good: A survey of challenges, opportunities, and responsible deployment.arXiv preprint arXiv:2505.22327, 2025

    arXiv 2025

  3. [3]

    Current applications and challenges in large language models for patient care: a systematic review.Communications Medicine, 5(1):26, 2025

    Felix Busch, Lena Hoffmann, Christopher Rueger, Elon HC van Dijk, Rawen Kader, Esteban Ortiz-Prado, Marcus R Makowski, Luca Saba, Martin Hadamitzky, Jakob Nikolas Kather, et al. Current applications and challenges in large language models for patient care: a systematic review.Communications Medicine, 5(1):26, 2025

    work page 2025

  4. [4]

    A survey on large language models with some insights on their capabilities and limitations.arXiv preprint arXiv:2501.04040, 2025

    Andrea Matarazzo and Riccardo Torlone. A survey on large language models with some insights on their capabilities and limitations.arXiv preprint arXiv:2501.04040, 2025

    arXiv 2025

  5. [5]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  6. [6]

    Reinforcement learning enhanced llms: A survey.arXiv preprint arXiv:2412.10400, 2024

    Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy. Reinforcement learning enhanced llms: A survey.arXiv preprint arXiv:2412.10400, 2024

    arXiv 2024

  7. [7]

    Llms for explainable ai: A comprehensive survey.arXiv preprint arXiv:2504.00125, 2025

    Ahsan Bilal, David Ebert, and Beiyu Lin. Llms for explainable ai: A comprehensive survey.arXiv preprint arXiv:2504.00125, 2025

    arXiv 2025

  8. [8]

    Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    work page 2024

Show all 56 references
  1. [9]

    Understand what llm needs: Dual preference alignment for retrieval-augmented generation

    Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Ji-Rong Wen, and Zhicheng Dou. Understand what llm needs: Dual preference alignment for retrieval-augmented generation. InProceedings of the ACM on Web Conference 2025, pages 4206–4225, 2025

    2025

  2. [10]

    Chain of preference optimization: Improving chain-of-thought reasoning in llms.Advances in Neural Information Processing Systems, 37:333–356, 2024

    Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thought reasoning in llms.Advances in Neural Information Processing Systems, 37:333–356, 2024

    2024

  3. [11]

    Context-dpo: Aligning language models for context-faithfulness.arXiv preprint arXiv:2412.15280, 2024

    Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, et al. Context-dpo: Aligning language models for context-faithfulness.arXiv preprint arXiv:2412.15280, 2024

    2024 arXiv

  4. [12]

    Pa-rag: Rag alignment via multi-perspective preference optimization.arXiv preprint arXiv:2412.14510, 2024

    Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, and Ming Gao. Pa-rag: Rag alignment via multi-perspective preference optimization.arXiv preprint arXiv:2412.14510, 2024

    2024 arXiv

  5. [13]

    Knowpo: Knowledge-aware preference optimization for controllable knowledge selection in retrieval- augmented language models

    Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, and Yasha Wang. Knowpo: Knowledge-aware preference optimization for controllable knowledge selection in retrieval- augmented language models. InProceedings of the AAAI Conference on Artific...

    2025

  6. [14]

    Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37:116617–116637, 2024

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37:116617–116637, 2024

    2024

  7. [15]

    Self-training with direct preference optimization improves chain-of- thought reasoning.arXiv preprint arXiv:2407.18248, 2024

    Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization improves chain-of- thought reasoning.arXiv preprint arXiv:2407.18248, 2024

    2024 arXiv

  8. [16]

    External knowledge integration in large language models: A survey on methods, challenges, and future directions.arXiv preprint arXiv:2403.11181, 2024

    Ishita Yadav, Simon Schindler, David Peters, and Roman Klinger. External knowledge integration in large language models: A survey on methods, challenges, and future directions.arXiv preprint arXiv:2403.11181, 2024

    2024 arXiv

  9. [17]

    A survey on knowledge-oriented retrieval-augmented generation.arXiv preprint arXiv:2503.10677, 2025

    Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, et al. A survey on knowledge-oriented retrieval-augmented generation.arXiv preprint arXiv:2503.10677, 2025

    2025 arXiv

  10. [18]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  11. [19]

    A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025. 10 RACE-Align

    2025 arXiv

  12. [20]

    A survey on personalized alignment–the missing piece for large language models in real-world applications.arXiv preprint arXiv:2503.17003, 2025

    Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized alignment–the missing piece for large language models in real-world applications.arXiv preprint arXiv:2503.17003, 2025

    2025 arXiv

  13. [21]

    Mm-rlhf: The next step forward in multimodal llm alignment.arXiv preprint arXiv:2502.10391, 2025

    Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal llm alignment.arXiv preprint arXiv:2502.10391, 2025

    2025 arXiv

  14. [22]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252, 2024

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models.arXiv preprint arXiv:2410.18252, 2024

    2024 arXiv

  15. [23]

    Rlhs: Mitigating misalign- ment in rlhf with hindsight simulation.arXiv preprint arXiv:2501.08617, 2025

    Kaiqu Liang, Haimin Hu, Ryan Liu, Thomas L Griffiths, and Jaime Fernández Fisac. Rlhs: Mitigating misalign- ment in rlhf with hindsight simulation.arXiv preprint arXiv:2501.08617, 2025

    2025 arXiv

  16. [24]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  17. [25]

    Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046, 2024

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.arXiv preprint arXiv:2405.21046, 2024

    2024 arXiv

  18. [26]

    M. Xu, Z. Chen, Z. Liu, T. Zhao, and D. Yu. Retrieval-dpo: Retrieval-augmented preference optimization with non-paired preference data. OpenReview, 2024

    2024

  19. [27]

    Safer-instruct: Aligning language models with automated preference data

    Taiwei Shi, Kai Chen, and Jieyu Zhao. Safer-instruct: Aligning language models with automated preference data. arXiv preprint arXiv:2311.08685, 2023

    2023 arXiv

  20. [28]

    Self-boosting large language models with synthetic preference data.arXiv preprint arXiv:2410.06961, 2024

    Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language models with synthetic preference data.arXiv preprint arXiv:2410.06961, 2024

    2024 arXiv

  21. [29]

    A comprehensive review of large language models: issues and solutions in learning environments.Discover Sustainability, 6(1):27, 2025

    Tariq Shahzad, Tehseen Mazhar, Muhammad Usman Tariq, Wasim Ahmad, Khmaies Ouahada, and Habib Hamam. A comprehensive review of large language models: issues and solutions in learning environments.Discover Sustainability, 6(1):27, 2025

    2025

  22. [30]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  23. [31]

    Zhang and J

    Y . Zhang and J. Wang. Retrieval-augmented generation: Advances and challenges in 2025.Journal of Artificial Intelligence Research, 62(3):456–478, 2025

    2025

  24. [32]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2(1), 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2(1), 2023

    2023 arXiv

  25. [33]

    Siru Liu, Allison B McCoy, and Adam Wright. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines.Journal of the American Medical Informatics Association, page oca...

    2025

  26. [34]

    Wang and Y

    H. Wang and Y . Shi. A survey on knowledge graph and retrieval-augmented generation for enhanced language model reasoning.Academic Journal of Science and Technology, 14(1):227–235, 2025

    2025

  27. [35]

    Feng and L

    T. Feng and L. He. Rgr-kbqa: Generating question answering logical forms using knowledge graph enhanced large language models. InProceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics, 2025

    2025

  28. [36]

    Retrieval-augmented generation with graphs (graphrag)

    Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309, 2024

    2024 arXiv

  29. [37]

    Synthetic data generation & multi-step rl for reasoning & tool use.arXiv preprint arXiv:2504.04736, 2025

    Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D Manning. Synthetic data generation & multi-step rl for reasoning & tool use.arXiv preprint arXiv:2504.04736, 2025

    2025 arXiv

  30. [38]

    Enhancing chain of thought prompting in large language models via reasoning patterns

    Yufeng Zhang, Xuepeng Wang, Lingxiang Wu, and Jinqiao Wang. Enhancing chain of thought prompting in large language models via reasoning patterns. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25985–25993, 2025

    2025

  31. [39]

    Transformers provably solve parity efficiently with chain of thought.arXiv preprint arXiv:2410.08633, 2024

    Juno Kim and Taiji Suzuki. Transformers provably solve parity efficiently with chain of thought.arXiv preprint arXiv:2410.08633, 2024

    2024 arXiv

  32. [40]

    Chain of draft: Thinking faster by writing less.arXiv preprint arXiv:2502.18600, 2025

    Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less.arXiv preprint arXiv:2502.18600, 2025. 11 RACE-Align

    2025 arXiv

  33. [41]

    Tool learning with large language models: A survey.Frontiers of Computer Science, 19(8):198343, 2025

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey.Frontiers of Computer Science, 19(8):198343, 2025

    2025

  34. [42]

    Making large language models better reasoners with alignment.arXiv preprint arXiv:2309.02144, 2023

    Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment.arXiv preprint arXiv:2309.02144, 2023

    2023 arXiv

  35. [43]

    Port: Preference optimization on reasoning traces.arXiv preprint arXiv:2406.16061, 2024

    Salem Lahlou, Abdalgader Abubaker, and Hakim Hacid. Port: Preference optimization on reasoning traces.arXiv preprint arXiv:2406.16061, 2024

    2024 arXiv

  36. [44]

    Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676, 2024

    Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676, 2024

    2024 arXiv

  37. [45]

    Preference tree optimization: Enhancing goal- oriented dialogue with look-ahead simulations

    Lior Baruch, Moshe Butman, Kfir Bar, and Doron Friedman. Preference tree optimization: Enhancing goal- oriented dialogue with look-ahead simulations. InScaling Self-Improving Foundation Models without Human Supervision

  38. [46]

    Large language models in traditional chinese medicine: A scoping review.Journal of Evidence-Based Medicine, 18(1):e12658, 2025

    Yaxuan Ren, Xufei Luo, Ye Wang, Haodong Li, Hairong Zhang, Zeming Li, Honghao Lai, Xuanlin Li, Long Ge, Janne Estill, et al. Large language models in traditional chinese medicine: A scoping review.Journal of Evidence-Based Medicine, 18(1):e12658, 2025

    2025

  39. [47]

    Tcmchat: A generative large language model for traditional chinese medicine.Pharmacological Research, 210:107530, 2024

    Yizheng Dai, Xin Shao, Jinlu Zhang, Yulong Chen, Qian Chen, Jie Liao, Fei Chi, Junhua Zhang, and Xiaohui Fan. Tcmchat: A generative large language model for traditional chinese medicine.Pharmacological Research, 210:107530, 2024

    2024

  40. [48]

    Biancang: A traditional chinese medicine large language model.arXiv preprint arXiv:2411.11027, 2024

    Sibo Wei, Xueping Peng, Yi-fei Wang, Jiasheng Si, Weiyu Zhang, Wenpeng Lu, Xiaoming Wu, and Yinglong Wang. Biancang: A traditional chinese medicine large language model.arXiv preprint arXiv:2411.11027, 2024

    2024

  41. [49]

    Qibo: A large language model for traditional chinese medicine.arXiv preprint arXiv:2403.16056, 2024

    Heyi Zhang, Xin Wang, Zhaopeng Meng, Zhe Chen, Pengwei Zhuang, Yongzhe Jia, Dawei Xu, and Wenbin Guo. Qibo: A large language model for traditional chinese medicine.arXiv preprint arXiv:2403.16056, 2024

    2024 arXiv

  42. [50]

    Ai-powered lawyering: Ai reasoning models, retrieval augmented generation, and the future of legal practice

    Daniel Schwarcz, Sam Manning, Patrick Barry, David R Cleveland, JJ Prescott, and Beverly Rich. Ai-powered lawyering: Ai reasoning models, retrieval augmented generation, and the future of legal practice. 2025

    2025

  43. [51]

    A comprehensive review on financial explainable ai.Artificial Intelligence Review, 58(6):1–49, 2025

    Wei Jie Yeo, Wihan Van Der Heever, Rui Mao, Erik Cambria, Ranjan Satapathy, and Gianmarco Mengaldo. A comprehensive review on financial explainable ai.Artificial Intelligence Review, 58(6):1–49, 2025

    2025

  44. [52]

    Findings of the association for computational linguistics: Eacl 2024

    L Tu, J Qu, S Yavuz, S Joty, W Liu, C Xiong, and Y Zhou. Findings of the association for computational linguistics: Eacl 2024. InAssociation for Computational Linguistics, pages 1278–1294, 2024

    2024

  45. [53]

    Aligning llms to ask good questions a case study in clinical reasoning.arXiv preprint arXiv:2502.14860, 2025

    Shuyue Stella Li, Jimin Mun, Faeze Brahman, Jonathan S Ilgen, Yulia Tsvetkov, and Maarten Sap. Aligning llms to ask good questions a case study in clinical reasoning.arXiv preprint arXiv:2502.14860, 2025

    2025 arXiv

  46. [54]

    Shennong-tcm: A traditional chinese medicine large language model

    Wenjing Yue Wei Zhu and Xiaoling Wang. Shennong-tcm: A traditional chinese medicine large language model. https://github.com/michael-wzhu/ShenNong-TCM-LLM, 2023

    2023

  47. [55]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  48. [56]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 12

    2002

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.