Pith. sign in

REVIEW 4 major objections 6 minor 85 references

Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a counterfactual-trained fact-checker plus GRPO reinforcement learning raises reasoning-step factuality by up to 49.9 points.

desk verdict The 49.9% factuality gain is measured with the same classifier used as the RL reward, and the paper's own case study shows the enhanced model still making the same medical errors. read the letter →

arxiv 2507.22940 v2 pith:4NXQMYKO submitted 2025-07-25 cs.CL cs.AI

Rui Jiao , Yue Zhang , Jinku Li This is my paper · ORCID
classification cs.CLcs.AI
keywords reasoningfactualitychain-of-thoughtcounterfactualaugmentationGRPOreinforcementlearningfact-checkingclassifiermechanisticinterpretabilityLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the visible intermediate reasoning steps of large language models are pervasively factually unreliable, even when the final answer is correct, and that this can be fixed. The proposed RELIANCE framework trains a fact-checking classifier on counterfactually corrupted reasoning chains, then uses that classifier as a reward in a GRPO reinforcement loop. On ten models, the authors report that improving reasoning-step factuality by up to 49.90 percentage points does not degrade final-answer performance on Math-500, AIME-2024, and GPQA. If right, this redirects LLM evaluation and training away from final answers and toward the reasoning users actually see.

What carries the argument

The load-bearing object is the fact-checking classifier: a small LoRA-adapted language model trained with supervised fine-tuning on 38,539 counterfactually augmented reasoning chains, where entities are replaced by same-type entities to create plausible but false statements. This classifier is then reused as the factual reward inside a GRPO reinforcement learning objective, alongside semantic, format, and length rewards. The mechanism works by making factual consistency a per-step optimization signal rather than a final-output property. The activation trajectory metrics (mean step distance, angular deviation, and adjacent-step cosine similarity) serve as the interpretability lens that connects training-induced behavior changes to internal representational changes.

What would settle it

Take a held-out set of naturally occurring reasoning errors (human-annotated, not entity-substituted), and compare the fact-checker's verdicts and the RELIANCE-trained model's step accuracy against human labels. If the classifier's agreement with humans on natural errors is no better than chance, or if the enhanced model's outputs are rated by blind experts as more evasive rather than more accurate, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that reasoning-step factuality is a measurable, trainable property distinct from final-answer accuracy. The authors build a binary classifier that labels each reasoning step as factually consistent or not, training it on chain-of-thought data where named entities have been replaced by same-type entities to create subtle counterfactual errors; the best LoRA-tuned version reaches 97.20% accuracy on their test set. They then feed this classifier's per-step verdicts into GRPO as one of four rewards (factual, semantic-similarity, format, length), reporting that the resulting policy improves step factuality from 42.20% to 92.10% on the weakest small model and from 68.53% to 82.65% on a 1.5B model, while extractive-match scores on standard benchmarks stay flat or improve. Mechanistic analysis of hidden-state activations shows that enhanced models have shorter step-to-step trajectory distances, smaller rotation angles, and higher adjacent-step cosine similarity, which the paper interprets as more coherent and semantically grounded reasoning.

Load-bearing premise

The whole result rests on the fact-checking classifier being a valid measure of reasoning-step factuality; if the entity-substitution training makes it blind to other error types, or if the policy learns to fool it, then the reported 49.90-point gain is not genuine factual improvement.

Editorial extensions

If this is right

  • Reasoning chains become auditable: each step can be scored by a fact-checking classifier, so 'correct answer but wrong thinking' becomes a detectable failure mode rather than an invisible one.
  • Small models can be brought closer to large-model factuality: the 0.5B model jumps from 42.20% to 92.10% step factuality after RELIANCE training, suggesting parameter-efficient RL is a viable reliability route.
  • RL rewards for reasoning quality can be multi-dimensional without sacrificing benchmark accuracy; Math-500, AIME-2024, GPQA, and LiveCodeBench scores are maintained or slightly improved.
  • Activation trajectory patterns (reduced divergence, higher step coherence) correlate with factual correctness, offering observable internal signatures of trustworthy reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the fact-checker is trained only on entity-substitution counterfactuals, it may miss errors that do not involve named entities, such as wrong causal links or miscalculations; a natural extension is to generate counterfactuals by corrupting relations, quantities, and logical connectives.
  • Editorial inference: if the classifier becomes the optimization target, the policy could learn to game it by producing evasive, hedged reasoning that the classifier labels 'true' while a human would call it unhelpful; the safety case study hints at this trade-off.
  • Editorial inference: the activation 'aha moment' signature could be exploited as a training signal or regularizer to encourage epistemic caution and step reconsideration, beyond what the current four-reward design achieves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces RELIANCE, a framework for detecting and improving factual accuracy in the intermediate reasoning steps of LLMs. It comprises three components: (1) a fact-checking classifier trained on counterfactually augmented (entity-substituted) reasoning chains; (2) a GRPO-based reinforcement learning procedure with multi-dimensional rewards combining factuality, semantic similarity, format compliance, and length constraints; and (3) a mechanistic interpretability analysis of hidden-state activations across reasoning steps. The authors evaluate reasoning-step factuality for ten white-box and black-box models, report that even strong models achieve only ~82% step-level factuality, and claim that RELIANCE improves reasoning-step factuality by up to 49.90 percentage points while preserving final-answer performance on Math-500, AIME-2024, GPQA, and LiveCodeBench. The central claim is that the RL procedure genuinely increases the factual correctness of intermediate reasoning steps.

Significance. The problem addressed is important and timely: visible reasoning chains that are fluent but factually flawed can mislead users in high-stakes settings, and most existing evaluations focus only on final answers. The paper's pipeline is broad and touches on a real gap. However, the core result is not supported as presented. The evaluation metric is identical to the reward used for training, the fact-checker is validated only on synthetic entity-substitution data, and the paper's own case study shows the enhanced model reproducing the same clinically dangerous errors it claims to remove. These issues are load-bearing, so the contribution as stated is not established. The mechanistic interpretability analysis is descriptive and may interest specialists, but it does not provide an independent check on factuality. The paper would need a substantially redesigned evaluation to support its claims.

major comments (4)
  1. [Section 4.4 vs. Eq. (9)] The primary evaluation metric, Reasoning Chain Steps Factuality Accuracy (SFA), is defined in Section 4.4 as Acc(T) = 1/n * sum_i f_theta(q, t_i), which is exactly the quantity optimized as the factual reward R_fact in Eq. (9) (with a threshold and indicator function). Because the policy is trained to maximize R_fact, the reported SFA improvements in Table 5, including the headline +49.90% gain, are expected by construction and do not constitute independent evidence that reasoning steps became more factually correct. To support the RQ2 claim, the authors must evaluate with a metric not used in training, such as a separately trained classifier, human annotation, or verification against an external knowledge base.
  2. [Section 6, Figure 6] The paper's own case study contradicts the claimed enhancement. The post-RELIANCE response to the pediatric morphine query still states that morphine is a stimulant, that a 1-year-old weighs 2.2 kg, and recommends 0.22 mg of morphine for vomiting, and Figure 6 explicitly annotates all of these as errors. The text describes 'remarkable improvements' and says RELIANCE 'prioritized safety by avoiding speculative dosing recommendations,' yet the displayed enhanced output contains the same clinically dangerous errors as the baseline. This internal inconsistency directly undercuts the central RQ2 claim.
  3. [Section 3.1, Table 3] The fact-checking classifier is trained and evaluated solely on synthetic entity-substitution counterfactuals. No evaluation is performed on real model-generated reasoning chains, and no human agreement study is reported. Without evidence that the classifier generalizes beyond this synthetic distribution, its validity as a measure of reasoning-step factuality is not established. When the same classifier is used both as the training reward and as the evaluation metric, the circularity is compounded and the reported gains become uninterpretable.
  4. [Section 5.2, Figure 2] The claim that RELIANCE maintains or improves final-answer performance is supported for only one of the four enhanced models (DeepSeek-R1-Distill-Qwen-1.5B) on the four benchmarks. The other three models are not shown, and no statistical significance tests or confidence intervals are provided. This is insufficient to support the general claim stated in the abstract and Section 5.2 that performance is 'maintained or slightly improved' across the approach.
minor comments (6)
  1. [Table 4] The reported 'Overall Acc.' for DeepSeek-R1-Distill-Qwen-7B is 71.46%, which is lower than every per-temperature value listed (70.37%–78.56%), so the aggregation appears inconsistent or misreported.
  2. [Equation (8)] The KL divergence formula appears to be written in the wrong direction: D_KL(pi_theta || pi_ref) should involve pi_theta/pi_ref - log(pi_theta/pi_ref) - 1, not the inverse ratio shown, which may also violate non-negativity.
  3. [Figure 1] The word 'False' is misspelled as 'Fasle' in Figure 1, and several labels are otherwise inconsistent with the text (e.g., 'Mechanical Interpretability' vs. 'mechanistic interpretability').
  4. [Figure 6] The model name 'FactCompass' appears in Figure 6 but is never defined in the text; it appears to be a leftover from another tool and should be removed or explained.
  5. [Table 5] The caption mentions shaded cells indicating relative improvement, but no shading is visible in the table as provided; this should be corrected for clarity.
  6. [Appendix A.3, Table 7] The ablation study reports that removing the format or length reward causes training to fail completely, and the fact-checking reward alone also fails. This suggests the framework is brittle and that the attribution of the reported gains to the factual reward is not straightforward; this point deserves fuller discussion in the main text.

Circularity Check

1 steps flagged · score 8.0 of 10

Headline factuality gain is measured with the same classifier used as the RL reward, so the 49.9% improvement is by construction.

  1. fitted input called prediction [Section 3.2, Eq. (9); Section 4.4, SFA definition; Appendix A.2]
    "Rfact(T,S|q) = 1/|T_valid| Σ_{t_i∈T_valid} I(P_{π*_θ}(t_i|q) > τ) (Eq. 9). ... Our primary metric is Reasoning Chain Steps Factuality Accuracy (SFA), defined as: Acc(T) = 1/n Σ_{i=1}^n f_θ(q,t_i) where f_θ(q,t_i) evaluates whether step segment t_i is factually consistent given question q (Section 4.4)."

    Eq. (9)'s factual reward and Section 4.4's SFA metric are the same classifier output: Rfact averages the indicator I(P_{π*_θ}(t_i|q)>τ) over steps, while Acc(T) averages f_θ(q,t_i), and Appendix A.2 defines f_θ as the fact-checking function returning True/False. GRPO (Eq. 6) directly maximizes this reward, so the reported +49.90% SFA improvement is the score of the objective being optimized; the increase is expected by construction and cannot by itself establish that reasoning became factually better. The classifier's own validity is established only on synthetic entity-substitution counterfactuals (Section 3.1/Table 3) with no human evaluation on real model-generated reasoning, closing the loop inside the paper's own data.

full rationale

The central circular step is the identity between the training reward and the headline evaluation metric. Eq. (9) defines Rfact as the average of the fact-checking classifier's thresholded probability over reasoning steps; Section 4.4 defines the Reasoning Chain Steps Factuality Accuracy metric Acc(T) as the average of the same classifier's binary judgment f_θ(q,t_i). Appendix A.2 confirms f_θ is the fact-checking function. Because the GRPO objective in Eq. (6) is maximized with Rfact as one of its rewards, the policy is explicitly trained to increase the exact quantity later reported as 'up to 49.90% improvement.' That makes the headline RQ2 gain a near-tautological consequence of the optimization setup: any successful optimization of Eq. (9) will raise Acc(T). The loop is closed by the fact that the classifier itself was trained on the authors' synthetic entity-substitution dataset (Section 3.1) and evaluated only on samples from the same distribution (Table 3); no human or external gold-standard evaluation of factuality on real reasoning chains is reported. The final-answer benchmarks (Math-500, AIME-2024, GPQA) are genuinely external and show no degradation, but they measure answer correctness, not reasoning-step factuality, so they do not independently confirm the SFA gain. The paper's own Figure 6 case study is also damaging: the RELIANCE-enhanced response for the pediatric morphine query still contains the same critical errors (morphine described as a stimulant; 1-year-old weight given as 2.2 kg; recommended 0.22 mg dose), which suggests the reward/evaluation loop can be gamed even on the paper's showcase example. The RQ1 landscape evaluation and RQ3 activation analysis are not themselves circular, and the related-work discussion is standard. The circularity is therefore partial but central: the primary empirical evidence for RQ2 reduces by construction to the fitted classifier used as both reward and metric. Score 8 rather than 10 because the optimization is genuine (the metric could in principle fail to rise) and because the framework contains non-circular components; but the headline factual-accuracy claim is not independently supported.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the fact-checking classifier's validity, which is assumed rather than demonstrated. The many hand-set reward parameters are free parameters that can absorb implementation details. No new entities are postulated.

free parameters (6)
  • factuality threshold tau = not reported
    Used in Eq. 9 to binarize the fact-checker probability into a reward; chosen by hand, no sensitivity analysis.
  • step length bounds Lmin and Lmax = not reported
    Define T_valid in Eq. 9; set empirically and affect which steps receive the factuality reward.
  • format reward constants alpha and beta = not reported
    Eq. 11; chosen by hand, no values or sensitivity analysis given.
  • length reward constants gamma and eta = not reported
    Eq. 12; chosen by hand, no values or sensitivity analysis given.
  • total length bounds Ltotal_min and Ltotal_max = not reported
    Eq. 12; determined empirically, values not reported.
  • KL regularization coefficient beta or lambda = not reported
    Eq. 6 and Eq. 8; the text mentions lambda but the equation uses beta, and the value is not reported.
assumptions (4)
  • domain assumption Replacing named entities with same-type entities produces factually wrong reasoning steps that are representative of real reasoning errors.
    Section 3.1 Phase I, used to build the training and test labels for the fact-checking classifier.
  • domain assumption The SFT fact-checking classifier generalizes from synthetic entity swaps to arbitrary factual errors in LLM reasoning chains.
    Used for evaluation in Tables 4 and 5 and as the reward in Eq. 9; no human validation on real model outputs is provided.
  • standard math The GRPO objective with reference policy regularization (Eqs. 6-8) is correct as written.
    Borrowed from DeepSeekMath [24], but the DKL formula in Eq. 8 appears inconsistent with the standard definition.
  • domain assumption Sentence-transformer cosine similarity is a valid proxy for final-answer correctness in the semantic reward (Eq. 10).
    Used to keep final answers aligned with references; no threshold value and no validation that cosine similarity tracks answer correctness are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes." pith.science (2026). https://pith.science/paper/4NXQMYKO

@misc{pith2026250722940,
  author       = {Pith},
  title        = {Pith review of: Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NXQMYKO}},
  note         = {Machine review of arXiv:2507.22940}
}
read the original abstract

We present a novel framework addressing a critical vulnerability in Large Language Models (LLMs): the prevalence of factual inaccuracies within intermediate reasoning steps despite correct final answers. This phenomenon poses substantial risks in high-stakes domains including healthcare, legal analysis, and scientific research, where erroneous yet confidently presented reasoning can mislead users into dangerous decisions. Our framework integrates three core components: (1) a specialized fact-checking classifier trained on counterfactually augmented data to detect subtle factual inconsistencies within reasoning chains; (2) an enhanced Group Relative Policy Optimization (GRPO) reinforcement learning approach that balances factuality, coherence, and structural correctness through multi-dimensional rewards; and (3) a mechanistic interpretability method examining how factuality improvements manifest in model activations during reasoning processes. Extensive evaluation across multi state-of-the-art models reveals concerning patterns: even leading models like Claude-3.7 and GPT-o1 demonstrate reasoning factual accuracy of only 81.93% and 82.57% respectively. Our approach significantly enhances factual robustness (up to 49.90% improvement) while maintaining or improving performance on challenging benchmarks including Math-500, AIME-2024, and GPQA. Furthermore, our neural activation-level analysis provides actionable insights into how factual enhancements reshape reasoning trajectories within model architectures, establishing foundations for future training methodologies that explicitly target factual robustness through activation-guided optimization.

Figures

Figures reproduced from arXiv: 2507.22940 by the authors.

Figure 1
Figure 1. The workflow of RELIANCE . TABLE 2: Example of our named entity replacement method used in the data augmentation pipeline. Original Entity Replacement Entity Type Adelaide Anne Procter Gene Wilder PERSON Original Text: Adelaide Anne Procter (30 October 1825 – 2 February 1864) was a significant English poet and philanthropist, known for her active role in social causes... Transformed Text: Gene Wilder (30 October 182… view at source ↗
Figure 2
Figure 2. Extractive match accuracy (%) comparison with [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Reasoning adjacent steps divergence analysis. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of feature representations across different layers using PCA (top row). Each column shows the latent [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: PCA comparison of neural activations across [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Case study analysis. dimensions. First, the model exhibited significantly greater epistemic caution, appropriately expressing uncertainty with phrases like “I’m not certain” and “this is outside my knowledge” when reasoning about specialized medical knowledge. Second, …
Figure 7
Figure 7. Figure 7: Training dynamics of the DeepSeek-R1-Distill-Qwen-1.5B model. (a) Training loss curve showing convergence; [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 29 canonical work pages

  1. [1]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, and J. K. et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020. [Online]. Ava...

    2020

  2. [2]

    Constitutional AI: harmlessness from AI feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, and J. K. et al., “Constitutional AI: harmlessness from AI feedback,” CoRR arXiv preprint, vol. abs/2212.08073, 2022. [Online]. Available: https: //doi.org/10.48550/arXiv.2212.08073

    arXiv doi:10.48550/arxiv.2212.08073 2022

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” CoRR arXiv preprint, vol. abs/2302.13971, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2302.13971

    arXiv doi:10.48550/arxiv.2302.13971 2023

  4. [4]

    Gemini: A family of highly capable multimodal models,

    R. Anil, S. Borgeaud, Y . Wu, J. Alayrac, and J. Y . et al., “Gemini: A family of highly capable multimodal models,” CoRR arXiv preprint , vol. abs/2312.11805, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2312.11805

    arXiv doi:10.48550/arxiv.2312.11805 2023

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  6. [6]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, and X. D. et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609 , 2023

    arXiv 2023

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, and S. Ma, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” CoRR, vol. abs/2501.12948,

    arXiv

  8. [8]

    Qwq-32b: Embracing the power of reinforcement learning,

    Q. Team, “Qwq-32b: Embracing the power of reinforcement learning,” March 2025. [Online]. Available: https://qwenlm.github. io/blog/qwq-32b/

    2025

Show all 85 references
  1. [9]

    Learning to reason with llms,

    OpenAI, “Learning to reason with llms,” https://openai.com/index/ learning-to-reason-with-llms/, 2024, 09-12

    2024

  2. [10]

    Claude 3.7 sonnet and claude code,

    Anthropic, “Claude 3.7 sonnet and claude code,” https://www. anthropic.com/news/claude-3-7-sonnet, 2025, 02-25

    2025

  3. [11]

    Large language models for disease diagnosis: A scoping review,

    S. Zhou, Z. Xu, M. Zhang, C. Xu, Y . Guo, Z. Zhan, S. Ding, J. Wang, K. Xu, Y . Fang, L. Xia, J. Yeung, D. Zha, G. B. Melton, M. Lin, and R. Zhang, “Large language models for disease diagnosis: A scoping review,” CoRR, vol. abs/2409.00097, 2024. [Online]. Available: https://do...

    2024 doi arXiv

  4. [12]

    Evaluating LLM - generated multimodal diagnosis from medical images and symptom analysis,

    D. P. Panagoulias, M. Virvou, and G. A. Tsihrintzis, “Evaluating LLM - generated multimodal diagnosis from medical images and symptom analysis,” CoRR, vol. abs/2402.01730, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.01730

    2024 doi arXiv

  5. [13]

    Towards robust legal reasoning: Harnessing logical llms in law,

    M. Kant, S. Nabi, M. Kant, R. Scharrer, M. Ma, and M. Nabi, “Towards robust legal reasoning: Harnessing logical llms in law,” CoRR, vol. abs/2502.17638, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2502.17638

    2025 doi arXiv

  6. [14]

    Investigating the shortcomings of llms in step-by-step legal reasoning,

    V . Mishra, B. Pathiraja, M. Parmar, S. Chidananda, and J. S. et al., “Investigating the shortcomings of llms in step-by-step legal reasoning,” CoRR, vol. abs/2502.05675, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2502.05675

    2025 doi arXiv

  7. [15]

    INVESTORBENCH: A benchmark for financial decision-making tasks with llm-based agent,

    H. Li, Y . Cao, Y . Yu, S. R. Javaji, Z. Deng, Y . He, Y . Jiang, Z. Zhu, K. Subbalakshmi, G. Xiong, J. Huang, L. Qian, X. Peng, Q. Xie, and J. W. Suchow, “INVESTORBENCH: A benchmark for financial decision-making tasks with llm-based agent,” CoRR, vol. abs/2412.18174, 2024. [O...

    2024 doi arXiv

  8. [16]

    Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline,

    K. Singh, S. Kaur, and C. Smiley, “Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline,” in Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF 2024, Brooklyn, NY, USA, November 14-17, 2024 . ACM, 2024, pp. 266...

    2024

  9. [17]

    Order matters in hallucination: Reasoning order as benchmark and reflexive prompting for large-language-models,

    Z. Xie, “Order matters in hallucination: Reasoning order as benchmark and reflexive prompting for large-language-models,” CoRR, vol. abs/2408.05093, 2024. [Online]. Available: https: //doi.org/10.48550/arXiv.2408.05093

    2024 doi arXiv

  10. [18]

    CMMLU: measuring massive multitask language understanding in chinese,

    H. Li, Y . Zhang, F. Koto, Y . Yang, H. Zhao, Y . Gong, N. Duan, and T. Baldwin, “CMMLU: measuring massive multitask language understanding in chinese,” in Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16,...

    2024 doi

  11. [19]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , S. Muresan, P. Nakov, an...

    2022 doi

  12. [20]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models,

    Y . Huang, Y . Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y . Zhang, J. Lei, Y . Fu, M. Sun, and J. He, “C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models,” in Advances in Neural Information Processing Systems 36: Annual Conferen...

    2023

  13. [21]

    Factuality enhanced language models for open-ended text generation,

    N. Lee, W. Ping, P. Xu, M. Patwary, P. Fung, M. Shoeybi, and B. Catanzaro, “Factuality enhanced language models for open-ended text generation,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 20...

    2022

  14. [22]

    SKILL: structured knowledge infusion for large language models,

    F. Moiseev, Z. Dong, E. Alfonseca, and M. Jaggi, “SKILL: structured knowledge infusion for large language models,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Sea...

    2022 doi

  15. [23]

    Contrastive learning reduces hallucination in conversations,

    W. Sun, Z. Shi, S. Gao, P. Ren, M. de Rijke, and Z. Ren, “Contrastive learning reduces hallucination in conversations,” in Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI ...

    2023

  16. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” CoRR, vol. abs/2402.03300,

    arXiv

  17. [25]

    Mechanistic interpretability for AI safety - A review,

    L. Bereska and E. Gavves, “Mechanistic interpretability for AI safety - A review,” CoRR, vol. abs/2404.14082, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.14082

    2024 doi arXiv

  18. [26]

    On the dangers of stochastic parrots: Can language models be too big?

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021 , M. C. Elis...

    2021

  19. [27]

    Sparks of artificial general intelligence: Early experiments with GPT- 4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with GPT- 4,” CoRR, vol. abs/2303.12712, 2023....

    2023 doi arXiv

  20. [28]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774,

    arXiv

  21. [29]

    Microsoft Bing: Get to know Bing,

    Microsoft, “Microsoft Bing: Get to know Bing,” 2023, accessed:

    2023

  22. [30]

    [Online]

    Google, “Bard,” 2023, accessed: 2023. [Online]. Available: https: //bard.google.com

    2023

  23. [31]

    CharacterAI, https://character.ai/, 2024

    2024

  24. [32]

    Poe, https://poe.com/, 2024

    2024

  25. [33]

    Chatlaw: Open-source legal large language model with integrated external knowledge bases,

    J. Cui, Z. Li, Y . Yan, B. Chen, and L. Yuan, “Chatlaw: Open-source legal large language model with integrated external knowledge bases,” CoRR, vol. abs/2306.16092, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2306.16092

    2023 doi arXiv

  26. [34]

    Available: https://www.microsoft.com/en-us/bing

    [Online]. Available: https://www.microsoft.com/en-us/bing

  27. [35]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” inAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information ...

    2022

  28. [36]

    Webgpt: Browser-assisted question-answering with human feedback,

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman, “Webgpt: Browser-assisted question-answering with human feedback,” CoRR, vol. ...

    2021 arXiv

  29. [37]

    Webcpm: Interactive web search for chinese long- form question answering,

    Y . Qin, Z. Cai, D. Jin, L. Yan, S. Liang, K. Zhu, Y . Lin, X. Han, N. Ding, H. Wang, R. Xie, F. Qi, Z. Liu, M. Sun, and J. Zhou, “Webcpm: Interactive web search for chinese long- form question answering,” in Proceedings of the 61st Annual Meeting of the Association for Comput...

    2023

  30. [38]

    The reversal curse: Llms trained on

    L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans, “The reversal curse: Llms trained on ”a is b” fail to learn ”b is a”,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Ope...

    2024

  31. [39]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 248:1–248:38,

  32. [40]

    Available: https://doi.org/10.1145/3571730

    [Online]. Available: https://doi.org/10.1145/3571730

    doi

  33. [41]

    MISGENDERED: limits of large language models in understanding pronouns,

    T. Hossain, S. Dev, and S. Singh, “MISGENDERED: limits of large language models in understanding pronouns,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, A. Rogers...

    2023 doi

  34. [42]

    Knowledge neurons in pretrained transformers,

    D. Dai, L. Dong, Y . Hao, Z. Sui, B. Chang, and F. Wei, “Knowledge neurons in pretrained transformers,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, S. Muresan, P...

    2022 doi

  35. [43]

    Locating and editing factual associations in GPT,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in GPT,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - Decem...

    2022

  36. [44]

    Improving factuality and reasoning in language models through multiagent debate,

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. [O...

    2024

  37. [45]

    Understanding catastrophic forgetting in language models via implicit inference,

    S. Kotha, J. M. Springer, and A. Raghunathan, “Understanding catastrophic forgetting in language models via implicit inference,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Ava...

    2024

  38. [46]

    Bias and fairness in large language models: A survey,

    I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N. K. Ahmed, “Bias and fairness in large language models: A survey,” CoRR, vol. abs/2309.00770,

    arXiv

  39. [47]

    Available: https://doi.org/10.48550/arXiv.2309.00770

    [Online]. Available: https://doi.org/10.48550/arXiv.2309.00770

    doi arXiv

  40. [48]

    Dola: Decoding by contrasting layers improves factuality in large language models,

    Y . Chuang, Y . Xie, H. Luo, Y . Kim, J. R. Glass, and P. He, “Dola: Decoding by contrasting layers improves factuality in large language models,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, ...

    2024

  41. [49]

    Improving language models by retrieving from trillions of tokens,

    S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, and K. M. et al., “Improving language models by retrieving from trillions of tokens,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , ser. Proceedings of Machine...

    2022

  42. [50]

    Internet-augmented language models through few-shot prompting for open-domain question answering,

    A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev, “Internet-augmented language models through few-shot prompting for open-domain question answering,” CoRR, vol. abs/2203.05115,

    arXiv

  43. [51]

    Rethinking with retrieval: Faithful large language model inference,

    H. He, H. Zhang, and D. Roth, “Rethinking with retrieval: Faithful large language model inference,” CoRR, vol. abs/2301.00303, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.00303

    2023 doi arXiv

  44. [52]

    LM vs LM: detecting factual errors via cross examination,

    R. Cohen, M. Hamri, M. Geva, and A. Globerson, “LM vs LM: detecting factual errors via cross examination,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, H. Bouamor, J. Pino, and K. Bali, E...

    2023 doi

  45. [53]

    Generate rather than retrieve: Large language models are strong context generators,

    W. Yu, D. Iter, S. Wang, Y . Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang, “Generate rather than retrieve: Large language models are strong context generators,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 202...

    2023

  46. [54]

    ”according to

    O. Weller, M. Marone, N. Weir, D. J. Lawrie, D. Khashabi, and B. V . Durme, “”according to . . . ”: Prompting language models improves quoting from pre-training data,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistic...

    2024

  47. [55]

    SAIL: search-augmented instruction learning,

    H. Luo, Y . Chuang, Y . Gong, T. Zhang, Y . Kim, X. Wu, D. Fox, H. Meng, and J. R. Glass, “SAIL: search-augmented instruction learning,” CoRR, vol. abs/2305.15225, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.15225

    2023 doi arXiv

  48. [56]

    Decoupled context processing for context augmented language modeling,

    Z. Li, R. Guo, and S. Kumar, “Decoupled context processing for context augmented language modeling,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - Dece...

    2022

  49. [57]

    G-MAP: general memory-augmented pre-trained language model for domain tasks,

    Z. Wan, Y . Yin, W. Zhang, J. Shi, L. Shang, G. Chen, X. Jiang, and Q. Liu, “G-MAP: general memory-augmented pre-trained language model for domain tasks,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United A...

    2022 doi

  50. [58]

    Mitigating language model hallucination with interactive question-knowledge alignment,

    S. Zhang, L. Pan, J. Zhao, and W. Y . Wang, “Mitigating language model hallucination with interactive question-knowledge alignment,” CoRR, vol. abs/2305.13669, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.13669

    2023 doi arXiv

  51. [59]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing ...

    2022

  52. [60]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions,

    H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2023 doi

  53. [61]

    Atlas: Few-shot learning with retrieval augmented language models,

    G. Izacard, P. S. H. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, “Atlas: Few-shot learning with retrieval augmented language models,” J. Mach. Learn. Res. , vol. 24, pp. 251:1–251:43, 2023. [Online]. Available: https...

    2023

  54. [62]

    REPLUG: retrieval-augmented black- box language models,

    W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W. Yih, “REPLUG: retrieval-augmented black- box language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

    2024

  55. [63]

    Lora learns less and forgets less,

    D. Biderman, J. J. G. Ortiz, J. P. Portes, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V . Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham, “Lora learns less and forgets less,” CoRR, vol. abs/2405.09673, 2024. [Online]. Available: https://doi.org/10.48550/arX...

    2024 doi arXiv

  56. [64]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  57. [65]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” CoRR arXiv preprint , vol. abs/1707.06347, 2017. [Online]. Available: http://arxiv.org/abs/ 1707.06347

    2017 arXiv

  58. [66]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” CoRR, vol. abs/2402.07927,

    arXiv

  59. [67]

    A survey on llm-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, Y . Wang, and J. Guo, “A survey on llm-as-a-judge,” CoRR, vol. abs/2411.15594, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2411.15594

    2024 doi arXiv

  60. [68]

    FLAIR: An easy-to-use framework for state-of-the- art NLP,

    A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. V ollgraf, “FLAIR: An easy-to-use framework for state-of-the- art NLP,” in NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, ...

    2019

  61. [69]

    Instruction tuning for large language models: A survey,

    S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang, “Instruction tuning for large language models: A survey,” CoRR, vol. abs/2308.10792, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2308.10792

    2023 doi

  62. [70]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. [Onli...

    2022

  63. [72]

    Lighteval: A lightweight framework for llm evaluation,

    C. Fourrier, N. Habib, H. Kydl ´ıˇcek, T. Wolf, and L. Tunstall, “Lighteval: A lightweight framework for llm evaluation,” 2023. [Online]. Available: https://github.com/huggingface/lighteval

    2023

  64. [73]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gon- zalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  65. [74]

    Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari, M. Zhang, A. A. Awan, C. Li, D. Li, E. Zheng, J. Rasley, S. Smith, O. Ruwase, and Y . He, “Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale,” CoRR arXiv preprint , vol. abs/2207.00032,

    arXiv

  66. [75]

    Available: https://doi.org/10.48550/arXiv.2402.07927

    [Online]. Available: https://doi.org/10.48550/arXiv.2402.07927

    doi arXiv

  67. [77]

    Open r1: A fully open reproduction of deepseek-r1,

    H. Face, “Open r1: A fully open reproduction of deepseek-r1,” January 2025. [Online]. Available: https://github.com/huggingface/ open-r1

    2025

  68. [78]

    Gemini 2.0 flash,

    google, “Gemini 2.0 flash,” https://deepmind.google/technologies/ gemini/flash//, 2025

    2025

  69. [79]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, Z. Luo, and Y . Ma, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” CoRR arXiv preprint , vol. abs/2403.13372, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.13372

    2024 doi arXiv

  70. [85]

    Available: https://doi.org/10.48550/arXiv.2207.00032

    [Online]. Available: https://doi.org/10.48550/arXiv.2207.00032

    doi arXiv

  71. [86]

    An analysis of encoder representations in transformer-based machine translation,

    A. Raganato and J. Tiedemann, “An analysis of encoder representations in transformer-based machine translation,” in Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP , BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018 , T. Linzen, G. Chr...

    2018 doi

  72. [623]

    Available: https://doi.org/10.1145/3442188.3445922

    [Online]. Available: https://doi.org/10.1145/3442188.3445922

  73. [2019]

    Available: http://arxiv.org/abs/1910.03771

    [Online]. Available: http://arxiv.org/abs/1910.03771

    1910 arXiv

  74. [2022]

    Available: https://doi.org/10.48550/arXiv.2203.05115

    [Online]. Available: https://doi.org/10.48550/arXiv.2203.05115

    doi arXiv

  75. [2023]

    Available: https://doi.org/10.48550/arXiv.2303.08774

    [Online]. Available: https://doi.org/10.48550/arXiv.2303.08774

    doi arXiv

  76. [2024]

    Available: https://doi.org/10.48550/arXiv.2402.03300

    [Online]. Available: https://doi.org/10.48550/arXiv.2402.03300

    doi arXiv

  77. [2025]

    Available: https://doi.org/10.48550/arXiv.2501.12948

    [Online]. Available: https://doi.org/10.48550/arXiv.2501.12948

    doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.