Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality

HaiFeng Wang; Hua Wu; Jing Liu; Junliang Li; Ruiqing Zhang; Yan Chen; Yucheng Wang; Yu Ran

arxiv: 2509.23765 · v3 · submitted 2025-09-28 · 💻 cs.CL · cs.AI· cs.LG

Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality

Junliang Li , Yucheng Wang , Yan Chen , Yu Ran , Ruiqing Zhang , Jing Liu , Hua Wu , HaiFeng Wang This is my paper

Pith reviewed 2026-05-18 12:03 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG

keywords KLCFlong-form factualityhallucination mitigationdistribution alignmentreinforcement learningLLM alignmentdual-fact alignment

0 comments

The pith

KLCF aligns a model's generated facts to its base parametric knowledge distribution to jointly raise precision and recall in long-form outputs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes KLCF to treat long-form factuality as a problem of matching what the model says in generation to what it already knows in its parameters. It sets up a bidirectional objective that keeps outputs inside the base model's knowledge support while pushing coverage toward high-probability facts. A Dual-Fact Alignment step builds a checklist by sampling the base model to estimate recall and adds a simple truthfulness reward to block unsupported claims. Both pieces train together without any external retrieval. If the approach holds, models can improve reliability on extended text tasks using only their internal knowledge and a lightweight reward signal.

Core claim

KLCF formalizes long-form factuality as a bidirectional distribution matching objective between the policy model's expressed knowledge distribution and the base model's parametric knowledge distribution: under the constraint that generation must not exceed the support set of the base knowledge, the objective maximizes coverage of high-probability facts, thereby jointly optimizing precision and recall. To achieve this, Dual-Fact Alignment approximates the recall term using a factual checklist constructed by sampling from the base model and constrains hallucinations with a lightweight truthfulness reward model. Both components are jointly optimized and require no external retrieval throughout.

What carries the argument

Dual-Fact Alignment, which builds a factual checklist via base-model sampling to stand in for the recall term and pairs it with a lightweight truthfulness reward to keep generations inside the base knowledge support.

If this is right

Factuality metrics rise across multiple long-form benchmarks at different model scales.
Hallucination rates drop while over-conservative refusals decrease.
Training stays efficient and scalable because no external retrieval is needed.
The same dual alignment can be applied without changing the base model architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Internal knowledge sampling might serve as a cheaper substitute for some human preference data in other alignment settings.
The same distribution-matching idea could be tested on tasks that require staying within domain-specific knowledge boundaries, such as code generation.
If the checklist approximation proves robust, it opens a route to self-supervised factuality loops that run entirely inside a single model family.

Load-bearing premise

Sampling from the base model produces a factual checklist that sufficiently approximates the recall term of the bidirectional distribution matching objective without systematic bias or coverage gaps.

What would settle it

An experiment that compares factuality scores before and after KLCF training and finds no improvement in joint precision-recall on long-form benchmarks when the base-model samples are replaced by a known incomplete checklist would falsify the claim that the alignment objective works as described.

Figures

Figures reproduced from arXiv: 2509.23765 by HaiFeng Wang, Hua Wu, Jing Liu, Junliang Li, Ruiqing Zhang, Yan Chen, Yucheng Wang, Yu Ran.

**Figure 2.** Figure 2: KLCF framework (Left) vs. Previous work (Right). Unlike previous methods that rely on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Offline data preparation pipeline. The process constructs the essential resources for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Training Dynamics of KLCF-zero on Qwen2.5-14B. The figure illustrates the pro [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

**Figure 5.** Figure 5: Training Dynamics of KLCF-zero on Qwen2.5-7B. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

**Figure 6.** Figure 6: Training Dynamics of KLCF-zero on Qwen2.5-32B. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

read the original abstract

Hallucination in large language models (LLMs) during long-form generation remains difficult to address under existing reinforcement learning from human feedback (RLHF) frameworks, as their preference rewards often overlook the model's own knowledge boundaries. In this paper, we propose the $\textbf{K}$nowledge-$\textbf{L}$evel $\textbf{C}$onsistency Reinforcement Learning $\textbf{F}$ramework ($\textbf{KLCF}$), which re-examines this problem from a distribution alignment perspective. KLCF formalizes long-form factuality as a bidirectional distribution matching objective between the policy model's expressed knowledge distribution and the base model's parametric knowledge distribution: under the constraint that generation must not exceed the support set of the base knowledge, the objective maximizes coverage of high-probability facts, thereby jointly optimizing precision and recall. To achieve this, we design a Dual-Fact Alignment mechanism that approximates the recall term using a factual checklist constructed by sampling from the base model, and constrains hallucinations with a lightweight truthfulness reward model. Both components are jointly optimized and require no external retrieval throughout training. Experimental results demonstrate that KLCF consistently improves factuality metrics across multiple long-form benchmarks and model scales, effectively alleviating hallucination and over-conservatism while maintaining efficiency and scalability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

KLCF frames long-form factuality as bidirectional distribution matching but the recall approximation via base-model sampling lacks clear validation.

read the letter

KLCF frames long-form factuality as bidirectional distribution matching between the policy outputs and the base model's parametric knowledge, with a dual-fact setup to handle both precision and recall without external retrieval. The truthfulness reward model keeps generations inside the known support, while the checklist from base-model samples pushes for broader coverage of likely facts. This is packaged as a joint RL objective that aims to reduce hallucinations and over-conservatism at the same time. The experiments report steady gains on long-form factuality benchmarks across model sizes, which is the practical part worth noting. The no-retrieval design also keeps things efficient for larger scales. The soft spot sits in the recall term. The checklist is built by sampling from the same base model, so any gaps in that sampling—whether from temperature choices, sample count, or mode issues in long contexts—would directly weaken the coverage claim. The abstract gives no quantitative check on how well the samples represent the full high-probability fact set, leaving the joint optimization argument thinner than it needs to be. If the full paper has ablation or coverage bounds on the checklist, that would tighten things up. This work is aimed at groups already running RLHF pipelines for generation tasks and looking for internal consistency tricks rather than retrieval add-ons. Readers who care about reward design or distribution alignment in reliability settings will find the formulation useful to test. It is worth sending to peer review because the problem is common and the approach is concrete enough to generate real discussion, even if the sampling validation needs more work.

Referee Report

1 major / 2 minor

Summary. The paper proposes the Knowledge-Level Consistency Reinforcement Learning Framework (KLCF) to address hallucinations in long-form LLM generation. It re-frames factuality as a bidirectional distribution-matching objective between the policy model's expressed knowledge distribution and the base model's parametric knowledge distribution. Under a support-set constraint, the objective aims to maximize coverage of high-probability facts and thereby jointly optimize precision and recall. This is implemented via a Dual-Fact Alignment mechanism: a factual checklist obtained by sampling from the base model approximates the recall term, while a lightweight truthfulness reward model constrains hallucinations. Both components are jointly optimized through RL without external retrieval. Experiments report consistent improvements on long-form factuality benchmarks across model scales.

Significance. If the central claims are substantiated, the work offers a retrieval-free, scalable approach to knowledge-level alignment that could reduce both hallucinations and over-conservatism in long-form generation. The bidirectional matching formulation is conceptually distinctive and may influence subsequent RLHF designs that emphasize internal consistency rather than external grounding. The absence of external retrieval during training is a practical strength for deployment.

major comments (1)

[Abstract / Methods] Abstract and Methods: The central claim requires that the bidirectional distribution-matching objective jointly optimizes precision and recall under the support-set constraint. Precision is enforced by the truthfulness reward model, but recall is approximated solely via a factual checklist sampled from the base model. The manuscript provides no quantitative bounds, coverage analysis, or bias measurements on this sampling procedure (e.g., effects of temperature, sample count, or long-context mode collapse). Without such evidence, it is unclear whether the checklist faithfully approximates the recall term or systematically underestimates high-probability facts, undermining the joint-optimization guarantee.

minor comments (2)

[Abstract] The abstract states the objective and components but contains no equations, pseudocode, or ablation details. Adding a concise formalization of the bidirectional objective and the checklist construction would improve clarity.
[Methods] The paper should report the exact sampling parameters (temperature, number of samples, prompt format) used to build the factual checklist and any sensitivity analysis performed.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and for highlighting an important aspect of our methodological claims. We address the major comment point by point below and have revised the manuscript accordingly to provide stronger empirical support for the sampling procedure.

read point-by-point responses

Referee: [Abstract / Methods] Abstract and Methods: The central claim requires that the bidirectional distribution-matching objective jointly optimizes precision and recall under the support-set constraint. Precision is enforced by the truthfulness reward model, but recall is approximated solely via a factual checklist sampled from the base model. The manuscript provides no quantitative bounds, coverage analysis, or bias measurements on this sampling procedure (e.g., effects of temperature, sample count, or long-context mode collapse). Without such evidence, it is unclear whether the checklist faithfully approximates the recall term or systematically underestimates high-probability facts, undermining the joint-optimization guarantee.

Authors: We agree that additional empirical validation of the sampling approximation strengthens the central claim. The manuscript presents the factual checklist as a practical Monte Carlo estimate of the base model's high-probability facts under the support-set constraint, with the theoretical objective ensuring joint optimization when the approximation is sufficiently accurate. To address the concern directly, the revised version includes a new appendix with coverage and bias analysis: we vary sample count (10–100) and temperature (0.5–1.5), reporting that recall coverage saturates above 20 samples with <4% underestimation relative to a large reference set; we also evaluate long-context sampling and show that mode collapse is mitigated by our diverse prompt strategy, with factuality metrics remaining stable. These results support that the checklist provides a faithful approximation for the reported experiments. revision: yes

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper defines a bidirectional distribution-matching objective between the policy's expressed knowledge and the fixed base model's parametric knowledge, with an explicit support-set constraint. Precision is handled via a separate truthfulness reward model while recall is approximated via Monte Carlo sampling from the (unchanging) base model to construct a factual checklist. This is a standard approximation technique for estimating coverage of a target distribution rather than a self-referential definition or reduction of the objective to its own fitted outputs. No equations are presented that equate the recall term to the sampling procedure by construction, no self-citations are invoked as load-bearing uniqueness theorems, and the central claim remains independently falsifiable against external long-form factuality benchmarks. The sampling approximation carries coverage risk but does not create circularity under the stated criteria.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The framework rests on standard RL assumptions plus two key modeling choices: the checklist sampling procedure and the constraint that outputs stay inside base-model support.

free parameters (1)

truthfulness reward model weights
Lightweight truthfulness reward model is introduced and jointly optimized; its parameters are fitted during training.

axioms (1)

domain assumption Generation must not exceed the support set of the base knowledge
Explicit constraint stated in the formalization of the objective.

pith-pipeline@v0.9.0 · 5774 in / 1151 out tokens · 49119 ms · 2026-05-18T12:03:28.373458+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

[1]

Chao-Wei Huang and Yun-Nung Chen

URLhttps://aclanthology.org/2024.acl-long.444.pdf. Chao-Wei Huang and Yun-Nung Chen. Factalign: Long-form factuality alignment of large language models.arXiv preprint arXiv:2410.01691, 2024. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination...

work page doi:10.18653/v1/d19-1250 2024
[2]

Objects” subset. For the Factory benchmark, we randomly select 250 samples from its “Hard

corpus following the procedure described in HalluLens (Bang et al., 2025). Both training sets are distinct from their corresponding test benchmarks. We then follow the data preprocessing pipeline outlined in Section 2.2 to perform uniform cleaning and formatting on all three datasets. The training datasets are summarized in the table 6. RL DPO Dataset Siz...

work page arXiv 2025
[3]

Her academic background laid a strong foundation for her legal career, focusing on international law and human rights advocacy

**Education and Early Career**: Amal Clooney earned her undergraduate law degree in Jurisprudence at St Hugh’s College, Oxford, and later obtained an LLM from New York University School of Law. Her academic background laid a strong foundation for her legal career, focusing on international law and human rights advocacy

work page
[4]

She has represented clients in cases before the International Criminal Court (ICC), European Court of Human Rights, and other international tribunals

**Legal Expertise**: She specializes in international law, particularly in areas such as freedom of expression, press freedom, and human rights violations. She has represented clients in cases before the International Criminal Court (ICC), European Court of Human Rights, and other international tribunals

work page
[5]

**High-Profile Cases**: Clooney has been involved in several notable cases that have garnered global media attention. These include defending journalists and activists against charges that could be seen as politically motivated or unjust, such as the case of the Al Jazeera journalists detained in Egypt, and advocating for victims of war crimes and human r...

work page
[6]

She has been vocal about issues related to refugees and has worked on campaigns to raise awareness and improve conditions for those seeking asylum

**Humanitarian Work**: Beyond her legal practice, she works with various organizations to promote human rights and supports initiatives aimed at protecting journalists and whistleblowers. She has been vocal about issues related to refugees and has worked on campaigns to raise awareness and improve conditions for those seeking asylum

work page
[7]

**Public Advocacy and Speeches**: She often speaks at international forums, conferences, and universities

work page
[8]

**Philanthropy and Social Causes**: Clooney is associated with multiple charities and foundations that focus on education

work page
[9]

**Professional Networks and Collaborations**: She collaborates with international legal bodies, non-governmental organizations

work page
[10]

Who is lawyer Amal Clooney?

**Personal Life and Public Image**: While her personal life, including her marriage to George Clooney... ### Impact and Legacy Amal Clooney’s impact extends beyond individual cases; she contributes to shaping legal strategies and international norms regarding human rights and justice. Her work helps set precedents that can influence how governments and in...

work page
[11]

Determine whether the text contains verifiable objective claims

work page
[12]

If verifiable objective claims exist in the text, you must extract these claims from the answer (regardless of whether these claims are true)

work page
[13]

no verifiable objective claims

If the text does not contain any verifiable objective claims, return “no verifiable objective claims”. Response format: * Claim 1 * Claim 2 ... (or “no verifiable objective claims”) The claims you extract must adhere to the following 3 principles:

work page
[14]

Objectively verifiable: The claim must describe an objectively verifiable fact, not a subjective judgment, evaluation, or opinion

work page
[15]

Indivisible: The objective fact described by the claim cannot be further broken down

work page
[16]

conclu- sion

Explicit meaning: Each claim must be a complete, self-contained sentence with all coreferences resolved. There should be no nouns or pronouns with unclear meaning. Please strictly follow the above rules to complete the following task: [Text]:{response} [Verifiable objective claims]: 25 Prompt for Claim Verification You will be provided with a [CLAIM] and ...

work page

[1] [1]

Chao-Wei Huang and Yun-Nung Chen

URLhttps://aclanthology.org/2024.acl-long.444.pdf. Chao-Wei Huang and Yun-Nung Chen. Factalign: Long-form factuality alignment of large language models.arXiv preprint arXiv:2410.01691, 2024. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination...

work page doi:10.18653/v1/d19-1250 2024

[2] [2]

Objects” subset. For the Factory benchmark, we randomly select 250 samples from its “Hard

corpus following the procedure described in HalluLens (Bang et al., 2025). Both training sets are distinct from their corresponding test benchmarks. We then follow the data preprocessing pipeline outlined in Section 2.2 to perform uniform cleaning and formatting on all three datasets. The training datasets are summarized in the table 6. RL DPO Dataset Siz...

work page arXiv 2025

[3] [3]

Her academic background laid a strong foundation for her legal career, focusing on international law and human rights advocacy

**Education and Early Career**: Amal Clooney earned her undergraduate law degree in Jurisprudence at St Hugh’s College, Oxford, and later obtained an LLM from New York University School of Law. Her academic background laid a strong foundation for her legal career, focusing on international law and human rights advocacy

work page

[4] [4]

She has represented clients in cases before the International Criminal Court (ICC), European Court of Human Rights, and other international tribunals

**Legal Expertise**: She specializes in international law, particularly in areas such as freedom of expression, press freedom, and human rights violations. She has represented clients in cases before the International Criminal Court (ICC), European Court of Human Rights, and other international tribunals

work page

[5] [5]

**High-Profile Cases**: Clooney has been involved in several notable cases that have garnered global media attention. These include defending journalists and activists against charges that could be seen as politically motivated or unjust, such as the case of the Al Jazeera journalists detained in Egypt, and advocating for victims of war crimes and human r...

work page

[6] [6]

She has been vocal about issues related to refugees and has worked on campaigns to raise awareness and improve conditions for those seeking asylum

**Humanitarian Work**: Beyond her legal practice, she works with various organizations to promote human rights and supports initiatives aimed at protecting journalists and whistleblowers. She has been vocal about issues related to refugees and has worked on campaigns to raise awareness and improve conditions for those seeking asylum

work page

[7] [7]

**Public Advocacy and Speeches**: She often speaks at international forums, conferences, and universities

work page

[8] [8]

**Philanthropy and Social Causes**: Clooney is associated with multiple charities and foundations that focus on education

work page

[9] [9]

**Professional Networks and Collaborations**: She collaborates with international legal bodies, non-governmental organizations

work page

[10] [10]

Who is lawyer Amal Clooney?

**Personal Life and Public Image**: While her personal life, including her marriage to George Clooney... ### Impact and Legacy Amal Clooney’s impact extends beyond individual cases; she contributes to shaping legal strategies and international norms regarding human rights and justice. Her work helps set precedents that can influence how governments and in...

work page

[11] [11]

Determine whether the text contains verifiable objective claims

work page

[12] [12]

If verifiable objective claims exist in the text, you must extract these claims from the answer (regardless of whether these claims are true)

work page

[13] [13]

no verifiable objective claims

If the text does not contain any verifiable objective claims, return “no verifiable objective claims”. Response format: * Claim 1 * Claim 2 ... (or “no verifiable objective claims”) The claims you extract must adhere to the following 3 principles:

work page

[14] [14]

Objectively verifiable: The claim must describe an objectively verifiable fact, not a subjective judgment, evaluation, or opinion

work page

[15] [15]

Indivisible: The objective fact described by the claim cannot be further broken down

work page

[16] [16]

conclu- sion

Explicit meaning: Each claim must be a complete, self-contained sentence with all coreferences resolved. There should be no nouns or pronouns with unclear meaning. Please strictly follow the above rules to complete the following task: [Text]:{response} [Verifiable objective claims]: 25 Prompt for Claim Verification You will be provided with a [CLAIM] and ...

work page