Pith. sign in

REVIEW 4 major objections 5 minor 46 references

RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B evaluator trained with rule-based reinforcement learning on synthetic response rankings matches or beats prompting-based judges 10–100× larger on RAG faithfulness and correctness, explaining each score.

desk verdict Strong faithfulness results and a clever training setup, but correctness claims are overstated and the load-bearing synthetic ranking assumption is unvalidated. read the letter →

arxiv 2505.22430 v1 pith:RQTACAJE submitted 2025-05-28 cs.CL

Kun Li , Yunxiang Li , Tianhua Zhang , Hongyin Luo , Xixin Wu , James Glass , Helen Meng This is my paper
classification cs.CL
keywords retrieval-augmentedgenerationRAGevaluationLLM-as-a-judgerule-basedreinforcementlearningGRPOcontext-awaredecodingfaithfulnessandcorrectnessinterpretable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that evaluating retrieval-augmented generation (RAG) responses does not require a giant judge model. Its central claim is that a 7-billion-parameter evaluator, trained end-to-end with rule-based reinforcement learning on automatically synthesized preference pairs, agrees with human judgments on faithfulness (whether claims are grounded in the retrieved passages) and correctness (whether claims are grounded in the ground-truth answer) as well as or better than prompting-based baselines built on models with 10–100 times more parameters. The device that makes training label-free is a ranking-based reward: candidate responses are generated with Context-Aware Decoding so that their degree of grounding is controlled by a scalar $\alpha$, and the ground-truth preference order is read off from $\alpha$ directly. If the claim holds, compact and transparent evaluators can replace expensive black-box LLM judges for RAG, with every score accompanied by a claim-by-claim explanation and verbatim evidence.

What carries the argument

The mechanism that carries the argument is a ranking-based reinforcement-learning loop powered by synthetic preferences. Candidate responses $\{y_{\alpha}\}$ are generated with Context-Aware Decoding, $y_i \sim \mathrm{softmax}[(1+\alpha)P_{\mathrm{LLM}}(\cdot\mid q,c,y_{<i}) - \alpha P_{\mathrm{LLM}}(\cdot\mid q,y_{<i})]$, where the scalar $\alpha$ controls how strongly generation is conditioned on the reference $c$; the paper assumes a monotone ground-truth ranking, $\alpha_i > \alpha_j \Rightarrow y_{\alpha_i} \succ y_{\alpha_j}$ (Eq. 4), which yields preference labels with zero human annotation. The evaluator, a 7B instruction-tuned base model, is optimized with GRPO, a group-relative policy-optimization RL algorithm, under three outcome rewards: format (the trajectory must parse as JSON), evidence (cited spans must be near-verbatim from the reference), and accuracy (the support ratio $S(y) = \#\{\text{supported claims}\}/\#\{\text{claims}\}$ must reproduce the ground-truth ranking). A curriculum schedule raises the number of candidates to be ranked from 3 to 4 as training progresses, and the combined reward (Eq. 6) ties the evaluator's reward to producing correct relative judgments through an interpretable chain of reasoning.

What would settle it

Run the response-synthesis procedure on a fresh set of questions, sample candidates at several $\alpha$ values (for instance $\alpha \in \{0, -0.5, -1, -1.4\}$), and have human annotators rank them under the same protocol used for the paper's benchmarks; if a nontrivial share of pairs violates the predicted ordering (larger $\alpha$ preferred), the monotonicity in Eq. 4 that generates every training preference fails, and the model's agreement with human judgment could no longer be traced to a sound training signal. A second probe: feed the trained evaluator response pairs whose true grounding order is known but whose claim structure is adversarially manipulated (one answer fragmented into many trivial claims, the other merged into a single claim), and check whether the ranking still follows genuine supportiveness or merely the model's self-reported support flags.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a compact model can be trained to carry out claim decomposition, evidence extraction, and supportiveness judgment as one end-to-end reasoning pass, and that this training can be driven entirely by synthetic preference data with zero human annotation. On the WikiEval faithfulness benchmark the trained 7B evaluator reaches 1.000 best-case, 0.992 middle-case, and 0.984 worst-case agreement with human faithfulness judgments, and on the RAGChecker meta-evaluation of correctness it reaches Pearson $r = 0.501$, Spearman $\rho = 0.452$, and Kendall $\tau = 0.354$ correlations with five-level human preference labels; in both settings it is at or near the top of every compared method, including GPT-4o and 70B-parameter backbones, with the faithfulness differences significant at the 0.01 level. The authors argue that because the reward is ranking-based rather than pointwise, the evaluator learns a finer-grained, more discriminative assessment of responses, and because every trajectory is a JSON-formatted chain of atomic claims and verbatim evidence, each score carries an interpretable audit trail.

Load-bearing premise

The entire training signal rests on the assumption that a response generated with a larger context-conditioning weight $\alpha$ is always preferred to one generated with a smaller $\alpha$ (Eq. 4), a monotonicity the paper never checks against human judgment or external ground truth before using it as the reward label.

Editorial extensions

If this is right

  • Automated RAG evaluation stops being tied to huge, expensive judge models: a 7B evaluator trained once can score faithfulness and correctness at a fraction of the inference cost of GPT-4o or 70B prompting pipelines.
  • Every score carries an audit trail of atomic claims, verbatim grounding evidence, and support decisions, so users can verify why a response was judged faithful or unfaithful instead of trusting a black-box number.
  • The training recipe needs no human annotation and no process-level reward labels, so the same pipeline can be re-run for new domains or reference types wherever controllable response synthesis is available.
  • Because faithfulness and correctness collapse into the same support-ratio formula, one trained evaluator covers both evaluation dimensions, and the ranking objective combined with curriculum scaling sharpens its discrimination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the $\alpha$-monotonicity assumption transfers, the same synthesize-then-rank recipe could train compact judges for other quality dimensions, such as summarization consistency or citation accuracy, wherever a controllable decoding knob orders candidates by the property of interest.
  • Editorial inference: the accuracy reward is computed from the model's own emitted \texttt{is_supported} flags, so a model could in principle satisfy the ranking objective by decomposing claims favorably rather than by genuinely grounding them; adversarially restructured candidate answers would reveal how resistant the reward is to this.
  • Editorial inference: the correctness table shows the untrained 72B rule-guided variant scoring highest on some correlation coefficients, which leaves open how much of the gain comes from the rule-guided prompt and how much from RL itself; a scale-by-training sweep would separate the two contributions.
  • Editorial inference: because the reasoning trajectory doubles as the training interface and the explanation interface, the approach could be extended to output confidence or flagged low-evidence decisions for high-stakes deployments of RAG systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RAG-Zeval, a compact 7B-parameter evaluator for RAG faithfulness and correctness. The method trains a Qwen2.5-7B-Instruct model by rule-based reinforcement learning to produce an end-to-end JSON evaluation trajectory: it decomposes responses into atomic claims, extracts grounding evidence, and judges claim support. To avoid human annotation, the authors synthesize candidate responses with Context-Aware Decoding (CAD) for different values of the alpha parameter and assume a monotonic ordering: larger alpha implies a more supportable response (Eq. 4). A ranking-based accuracy reward then checks whether the model's own inferred scores reproduce this assumed ordering, supplemented by format and verbatim-evidence rewards. The authors evaluate on WikiEval for faithfulness and the RAGChecker meta-evaluation set for correctness, reporting that RAG-Zeval achieves the strongest correlation with human judgments and outperforms LLM-based baselines with 10-100x more parameters. The paper includes ablations on the ranking objective, curriculum learning, and rule-guided reasoning, plus a case study illustrating the model's interpretability.

Significance. If the central claims hold, the paper makes a valuable contribution: it demonstrates that a compact, interpretable evaluator can be trained with zero human annotation and achieve state-of-the-art alignment with human judgments on RAG evaluation. The strengths of the work are its external validation on two human-annotated benchmarks, detailed descriptions of the prompt and hyperparameters, and systematic ablations showing the importance of the full ranking objective and curriculum learning. The approach is also refreshingly concrete: the evaluation trajectory is machine-parseable JSON, the reward components are explicit, and the case study illustrates the model's reasoning. However, the training signal rests on an unvalidated monotonicity assumption (Eq. 4), and the accuracy reward is computed from the model's own emitted support judgments, raising the risk that the model is rewarded for self-consistency rather than genuine evaluation skill. In addition, the correctness results do not fully support the 'strongest correlation' claim: a 72B non-RL variant of the authors' own approach achieves higher correlations than the final 7B model on the correctness benchmark.

major comments (4)
  1. [Section 3.3.1, Eq. (4)] The assumption that alpha_i > alpha_j implies y_alpha_i is preferred to y_alpha_j is load-bearing but unvalidated. CAD shifts the entire token distribution toward the passage, and larger alpha can produce copying, repetition, or over-anchored text, while smaller (negative) alpha can produce fluent but not necessarily less supported text. This is an empirical claim, not a mathematical identity. Since all RL preferences and the SFT baseline derive from this ranking, a nontrivial violation rate would make the training labels unreliable. Please validate monotonicity on a sample of synthesized responses using human pairwise judgments or an external entailment model, and report per-adjacent-pair agreement (e.g., for alpha pairs (0,-0.5), (-0.5,-1), (-1,-1.4)).
  2. [Section 3.3.2, Eq. (5)] The accuracy reward compares the model's ranking S(y), computed from is_supported flags that the model itself emits, to the synthetic alpha-order. This rewards self-consistency with the alpha-order, not necessarily semantic support. The evidence reward only measures verbatim longest-common-substring overlap, so a model could learn to assign is_supported values to match the alpha ranking without genuinely judging entailment. The external benchmarks show transfer, but they do not establish that the training signal was correct. Please include a reward-hacking analysis, such as correlating the model's emitted is_supported decisions on training responses with an independent entailment labeler or with human annotations on a held-out sample.
  3. [Section 5, Table 2] The abstract and Section 5 claim that RAG-Zeval achieves the strongest correlation with human preference and outperforms LLMs with 10-100x more parameters. On the correctness benchmark, Table 2 shows that RAG-Zeval w/o RL with Qwen2.5-72B achieves higher correlations (Pearson 0.521, Spearman 0.482, Kendall 0.388) than the final RAG-Zeval 7B model (0.501, 0.452, 0.354). The claim is therefore true only for faithfulness, not for correctness. Please qualify the claim or provide an explicit justification for preferring the 7B model despite these lower correctness correlations.
  4. [Section 4.2 and Tables 1-5] The dagger marks in Tables 1 and 2 indicate statistical significance at the 0.01 level, but no statistical test, sample size, or multiple-comparison correction is described. Tables 3-5 report averages over five runs without error bars or significance tests. Many of the reported differences are small (e.g., 0.501 vs. 0.495 in Table 2), and the claim of dominance requires confidence intervals or paired significance tests. Please report the test procedure and, ideally, bootstrap confidence intervals for all reported correlations.
minor comments (5)
  1. [Section 3.3.1, Eq. (3)] The text says 'PLLM (∗ |q,c,r <i)' but Eq. (3) uses y_<i; the variable r appears to be a typo and should be y_<i.
  2. [Figure 1] The caption states 'The ground-truth ranking of y's depends on the value of alpha'; this should be phrased as an assumption, e.g., 'The assumed ranking of y's is defined by the value of alpha', to match Eq. (4).
  3. [Table 1 caption] The phrase 'Non- GPT results' should be 'Non-GPT results'.
  4. [Section 4.4] The variant 'RAG-Zeval w/o RL' is used in tables but is never explicitly defined in the main text; please state that it is the same rule-guided prompt used without reinforcement learning.
  5. [Appendix A.4] The sentence 'For correctness evaluation, not all baseline evaluation framework has a direct correctness metric' has a subject-verb agreement error; it should be 'not all baseline evaluation frameworks have a direct correctness metric'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is validated against independent human-annotated benchmarks, and the synthetic ranking assumption is an input premise rather than a derived result.

full rationale

The paper's central claim is that a 7B-parameter evaluator trained with rule-based RL on synthetic rankings matches or outperforms larger LLM judges in correlation with human judgments. This claim is tested on WikiEval and the RAGChecker meta-evaluation dataset, both of which contain independent human preference labels. The training signal does rely on Eq. (4), which asserts that larger context-aware-decoding alpha values imply strictly preferred responses, and the accuracy reward in Eq. (5) checks consistency with that assumed ranking. This is an empirical premise about the synthesis procedure, not a conclusion derived from the evaluation benchmarks. Because the final measured outcome is correlation with human judgments on held-out data, the result is not forced by the training objective. The paper's self-citations (e.g., Li et al., 2024, 2025) appear only in general related-work statements and are not load-bearing for the method or its evaluation. The limitations section candidly notes the need for further multilingual and domain-specific validation, and the ethical considerations acknowledge potential bias in synthetic training data. No circular step could be identified where a prediction reduces by construction to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method does not introduce new physical entities, but it relies on several hand-chosen parameters and untested domain assumptions, particularly the monotonicity of the synthetic ranking and the self-reported support labels used in the reward.

free parameters (5)
  • CAD parameter values alpha = {0, -0.5, -1, -1.4}
    Hand-chosen to create a spread of groundedness; no sensitivity analysis, and the assumed monotonic ranking is the sole training label.
  • Evidence span length threshold = 10 tokens
    Spans shorter than 10 tokens receive zero evidence reward; set by hand.
  • Reward weighting = 1.0 accuracy, 0.5*re evidence, -0.5 format penalty
    Combined reward in Eq. 6 is hand-designed and not tuned or ablated except for the ranking objective.
  • Curriculum schedule = 3 candidates in epoch 1, 4 in epoch 2
    Chosen for training; static 4-response was worse, but no systematic exploration.
  • RL hyperparameters = KL 0.015, lr 1e-6, rollout temp 1.0, sample 8
    Standard values; no reported sensitivity.
assumptions (4)
  • domain assumption Monotonic relationship between CAD parameter alpha and response groundedness (Eq. 4)
    The synthetic ground-truth rankings used as RL rewards are derived from this monotonicity; it is never checked per instance.
  • domain assumption The evaluator's own claim-level support flags are valid enough to compute the accuracy reward
    The RL accuracy reward in Eq. 5 uses is_supported values produced by the model itself, making the reward partially self-referential.
  • domain assumption Context-Aware Decoding with in-context learning approximates the required conditional distributions
    Response synthesis relies on estimating P_LLM(*|q,c) and P_LLM(*|q) via in-context prompts (Table 9).
  • domain assumption Longest common substring normalized by span length is a meaningful proxy for evidence grounding
    Evidence reward uses this lexical overlap measure, which favors verbatim copying over paraphrase.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning." pith.science (2026). https://pith.science/paper/RQTACAJE

@misc{pith2026250522430,
  author       = {Pith},
  title        = {Pith review of: RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQTACAJE}},
  note         = {Machine review of arXiv:2505.22430}
}
read the original abstract

Robust evaluation is critical for deploying trustworthy retrieval-augmented generation (RAG) systems. However, current LLM-based evaluation frameworks predominantly rely on directly prompting resource-intensive models with complex multi-stage prompts, underutilizing models' reasoning capabilities and introducing significant computational cost. In this paper, we present RAG-Zeval (RAG-Zero Evaluator), a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task. Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments with detailed explanation in one-pass. We introduce a ranking-based outcome reward mechanism, using preference judgments rather than absolute scores, to address the challenge of obtaining precise pointwise reward signals. To this end, we synthesize the ranking references by generating quality-controlled responses with zero human annotation. Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments and outperforming baselines that rely on LLMs with 10-100 times more parameters. Our approach also exhibits superior interpretability in response evaluation.

Figures

Figures reproduced from arXiv: 2505.22430 by the authors.

Figure 1
Figure 1. An overview of RAG-ZEval. We synthesize training data using Context-Aware Decoding. The complete prompt is presented in Fig.5. The ground-truth ranking of y’s depends on the value of α. erence context (e.g., ground-truth answer for cor￾rectness or retrieved passages for faithfulness). In a RAG setting, a response is considered 1) cor￾rect if the ground-truth answer supports the claims of the response, or 2) faithful… view at source ↗
Figure 2
Figure 2. The density distribution of the scores assigned by the faithfulness evaluators.The distribution of the faithful [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) shows the changes of decomposed claim [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Reward dynamics of RL training with dif [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The complete prompt used in training the evaluator. Given the current question, context, and [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Case study (Part-1): Human annotators judge Response B as significantly better than Response A for the [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Case study (Part-2): For the same question and responses (shown in Fig. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 13 canonical work pages

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Yoshua Bengio, J\' e r\^ o me Louradour, Ronan Collobert, and Jason Weston. 2009. https://doi.org/10.1145/1553374.1553380 Curriculum learning . In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 41–48, New York, NY, USA. Association for Computing Machinery

    arXiv 2009

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement lea...

    arXiv 2025

  4. [4]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. https://arxiv.org/abs/2405.06211 A survey on rag meeting llms: Towards retrieval-augmented large language models . Preprint, arXiv:2405.06211

    arXiv 2024

  5. [5]

    Ferrara, Ethan-Tonic, and O

    J. Ferrara, Ethan-Tonic, and O. M. Ozturk. 2024. The rag triad. https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

    work page 2024

  6. [6]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    arXiv 2024

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

    arXiv 2024

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://api.semanticscholar.org/CorpusID:274234014 A survey on llm-as-a-judge . ArXiv, abs/2411.15594

    arXiv 2024

Show all 46 references
  1. [9]

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. https://arxiv.org/abs/2501.04519 rstar-math: Small llms can master math reasoning with self-evolved deep thinking . Preprint, arXiv:2501.04519

    2025 arXiv

  2. [10]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. https://api.semanticscholar.org/CorpusID:127986954 The curious case of neural text degeneration . ArXiv, abs/1904.09751

    2019 arXiv

  3. [11]

    Alex-Răzvan Ispas, Charles-Elie Simon, Fabien Caspani, and Vincent Guigue. 2025. https://api.semanticscholar.org/CorpusID:277150863 Towards lighter and robust evaluation for retrieval augmented generation . The Next Frontier in Reliable AI": Workshop on ICLR 2025, abs/2503.16161

    2025 arXiv

  4. [12]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, an...

    2019

  5. [13]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  6. [14]

    Kun Li, Tianhua Zhang, Yunxiang Li, Hongyin Luo, Abdalla Moustafa, Xixin Wu, James Glass, and Helen Meng. 2025. https://arxiv.org/abs/2503.01695 Generate, discriminate, evolve: Enhancing context faithfulness via fine-grained sentence-level self-evolution . Preprint, arXiv:2503.01695

    2025 arXiv

  7. [15]

    Kun Li, Tianhua Zhang, Xixin Wu, Hongyin Luo, James Glass, and Helen Meng. 2024. https://arxiv.org/abs/2410.18415 Decoding on graphs: Faithful and sound reasoning on knowledge graphs through generation of well-formed chains . Preprint, arXiv:2410.18415

    2024 arXiv

  8. [16]

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. https://aclanthology.org/2023.acl-long.687/ Contrastive decoding: Open-ended text generation as optimization . In Annual Meeting of the Associatio...

    2022

  9. [17]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  10. [18]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  11. [19]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025 a . https://arxiv.org/abs/2503.20783 Understanding r1-zero-like training: A critical perspective . Preprint, arXiv:2503.20783

    2025 arXiv

  12. [20]

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025 b . https://arxiv.org/abs/2504.02495 Inference-time scaling for generalist reward modeling . Preprint, arXiv:2504.02495

    2025

  13. [21]

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025 c . https://api.semanticscholar.org/CorpusID:277510339 Inference-time scaling for generalist reward modeling

    2025

  14. [22]

    Ilya Loshchilov and Frank Hutter. 2017. https://api.semanticscholar.org/CorpusID:53592270 Decoupled weight decay regularization . In International Conference on Learning Representations

    2017

  15. [23]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.557 S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  16. [24]

    Taylor, and Peter Stone

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: a framework and survey. 21(1)

    2020

  17. [25]

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski S...

    2022 arXiv

  18. [26]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  19. [27]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  20. [28]

    Nicholas Pipitone and Ghita Houir Alami. 2024. https://api.semanticscholar.org/CorpusID:271909426 Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain . ArXiv, abs/2408.10343

    2024 arXiv

  21. [29]

    Qwen. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  22. [30]

    Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. https://www.amazon.science/publicati...

    2024

  23. [31]

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. https://doi.org/10.18653/v1/2024.naacl-long.20 ARES : An automated evaluation framework for retrieval-augmented generation systems . In Proceedings of the 2024 Conference of the North American Chapter o...

    2024 doi

  24. [32]

    ES Shahul, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2023. https://api.semanticscholar.org/CorpusID:263152733 Ragas: Automated evaluation of retrieval augmented generation . In Conference of the European Chapter of the Association for Computational Linguistics

    2023

  25. [33]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. 2024. https://api.semanticscholar.org/CorpusID:267412607 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . ArXiv, abs/2402.03300

    2024 arXiv

  26. [34]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    2024 arXiv

  27. [35]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Yih. 2023. https://aclanthology.org/2024.naacl-short.69/ Trusting your evidence: Hallucinate less with context-aware decoding . In North American Chapter of the Association for Computational Li...

    2023

  28. [36]

    Yixuan Tang and Yi Yang. 2024. https://arxiv.org/abs/2401.15391 Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries . Preprint, arXiv:2401.15391

    2024 arXiv

  29. [37]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  30. [38]

    Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. https://api.semanticscholar.org/CorpusID:269758033 Evaluation of retrieval-augmented generation: A survey . ArXiv, abs/2405.07437

    2024 arXiv

  31. [39]

    Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2025 a . https://doi.org/10.1007/978-981-96-1024-2_8 Evaluation of Retrieval-Augmented Generation: A Survey , page 102–120. Springer Nature Singapore

    2025 doi

  32. [40]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, and 16 others. 2025 b . https://arxiv.org/abs/250...

    2025 arXiv

  33. [41]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  34. [42]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  35. [43]

    Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. 2025. Medrag: Enhancing retrieval-augmented generation with knowledge graph-elicited reasoning for healthcare copilot. In Proceedings of the ACM on Web Conference 2025, pages 4442--4457

    2025

  36. [44]

    Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. 2025. https://arxiv.org/abs/2504.12328 A comprehensive survey of reward models: Taxonomy, applications, challenges, and future . Preprint, arXiv:2504.12328

    2025 arXiv

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.