Pith. sign in

REVIEW 4 major objections 3 minor 29 references

DecMetrics: Structured Claim Decomposition Scoring for Factually Consistent LLM Outputs

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DecMetrics introduces three automatic scores—coverage, correctness, and non-redundancy—for judging whether a claim decomposition is good enough to trust.

desk verdict The metric definitions are clear and the combination is new, but the paper's evidence that they measure decomposition quality is circular and the downstream results undercut the central claim. read the letter →

arxiv 2509.04483 v1 pith:DLIGNWWW submitted 2025-08-31 cs.CL cs.AI

Minghui Huang This is my paper
classification cs.CLcs.AI
keywords claimdecompositionfact-checkingatomicclaimsnaturallanguageinferencesemanticentropyreinforcementlearningLLMfactualitybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fact-checking systems that verify long-form AI output usually start by chopping the text into tiny atomic claims, and the quality of that chop determines the final verdict. The paper argues that this decomposition step is rarely evaluated, and a bad one can make a false text look true or a true text look false. To fix this, DecMetrics defines three automatic scores—Completeness, Correctness, and Semantic Entropy—which measure whether the atomic claims together cover the original claim, stay faithful to it, and avoid saying the same thing twice. The authors build synthetic training data from Wikipedia summaries, teach NLI models to compute the three scores, and then use those scores as a reward to train a compact decomposition model. Their result is a claim-decomposition benchmark, Claim2Atom, plus a small model that, according to the paper, produces decompositions competitive with much larger LLMs.

What carries the argument

The decomposition tree is the load-bearing structure. Wikipedia summaries are split recursively until each claim is non-splittable, producing a hierarchy of claims and atomic claims. Sibling subtrees yield training pairs: dropping leaves produces 'incomplete' negatives, claims paired with foreign nodes produce 'incorrect' negatives, and overlapping sibling claims produce 'redundant' positives for the Semantic Entropy classifier. The same hierarchy supplies the composite reward R = α·Completeness + β·Correctness + γ·Semantic_Entropy, which is used to train DecModel via PPO, and the filtered subtrees form DecData, the new component of Claim2Atom.

What would settle it

Take a random sample of claims from Claim2Atom, have several human annotators independently judge each decomposition on the three axes, and compare their labels with DecMetrics' scores. If agreement on any axis is no better than chance, the claim that these metrics measure decomposition quality fails. A more direct check: for Correctness, construct decompositions that humans judge 50/50 on whether an atomic claim was present in the original text, and see whether the NLI scorer's supported/unsupported boundary tracks the human boundary.

Watch

Extended reading notes

Core claim

DecMetrics frames decomposition quality as three entailment-based scores. Completeness is the entailment probability of the merged atomic claims given the original claim; Correctness is the fraction of atomic claims entailed by the original claim; Semantic Entropy clusters atomic claims that mutually entail each other and reports the entropy of that clustering, so that repeated paraphrases lower the score. The paper's central claim is that these three axes capture the ways decomposition goes wrong—omitted information, fabricated facts, and redundant overlap—and that they can be learned automatically. Training labels come from a synthetic pipeline that recursively decomposes Wikipedia summari

Load-bearing premise

The load-bearing premise is that the synthetic labels—produced by an LLM decomposition followed by an LLM reverse check—are correct characterisations of completeness, correctness, and independence; the paper provides no human annotation or inter-annotator agreement to establish that these labels match what human judges would call good decompositions.

Editorial extensions

If this is right

  • Fact-checking systems can now score their own decomposition step before trusting it, catching omissions, fabrications, and duplicates before they distort the final factuality score.
  • Because the reward is composite, the training recipe suggests that a small model can be pushed toward all three quality axes at once, offering a low-cost alternative to prompting huge LLMs.
  • Claim2Atom gives later work a single benchmark with three measured axes, so decomposition models can be compared on coverage and fidelity rather than on output format alone.
  • The metrics inherit all limits of NLI models; the paper shows off-the-shelf NLI models score poorly on its test set, which means task-specific fine-tuning is a necessary part of the pipeline.
  • The ablations imply the three metrics are complementary, not redundant: dropping any one component measurably reduces the score it controls.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its synthetic labels come from an LLM reverse check with no human annotation; a natural next test is whether human raters agree with DecMetrics on the same decompositions, since low agreement would mean the metrics are calibrated to LLM judgment rather than to what readers count as missing or invented.
  • Semantic Entropy, as defined, penalizes any mutual entailment between atomic claims, which could suppress legitimate elaboration; a testable refinement would weight redundancy by whether repeated content is load-bearing for verification.
  • The same decomposition-tree machinery could transfer to other structured text problems—for example, evaluating retrieval-augmented answers for coverage and hallucination—by substituting arbitrary source documents for Wikipedia summaries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes DecMetrics, three automatic metrics (COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY) intended to evaluate the quality of claim decomposition in LLM-generated text. The metrics are implemented with NLI-based models fine-tuned on synthetic data produced by a Wikipedia-based pipeline: Qwen3-32B decomposes claims and an LLM reverse-check labels the resulting subtrees. The same metrics are used as a reward signal in a three-stage PPO pipeline to train DecModel, a lightweight T5-based decomposition model. The paper also introduces Claim2Atom, a benchmark combining FActScore, WICE, and the new DecData. Experiments report high internal accuracy for DecMetrics on its own test set, competitive scores for DecModel on Claim2Atom, and downstream fact-checking results that show a decrease relative to baselines.

Significance. The idea of formalizing decomposition quality into structured reward components and training a compact model with them is interesting and could be useful if the metrics were valid. The paper releases code and data, and the three-stage SFT/reward/PPO recipe is clearly described. However, the central claim is not established: the metrics are validated only against the same synthetic label generator that created them, and the main downstream experiment shows that DecModel underperforms both GPT-4 decomposition and no decomposition. The contribution, as presented, is therefore not supported.

major comments (4)
  1. [§3.2, Table 1] The supported/unsupported labels for all three metrics are produced by the pipeline in Steps 3–4: Qwen3-32B decompositions and an LLM reverse-check (Appendix Fig. 7). The fine-tuned DecMetrics scorers are trained and tested on synthetic examples derived from this same pipeline (§3.2.2). Table 1's 99.14% accuracy therefore measures agreement with the label generator, not with any independent notion of decomposition quality. No human annotation, inter-annotator agreement, or correlation with existing quality judgments is reported. Since the PPO reward in §4.1 and the Claim2Atom evaluation in Table 3 inherit these labels, the central claim that the metrics assess quality is unsupported.
  2. [§6.2, Table 4] The downstream experiment fails to support the abstract's claim that the approach enhances fact-checking. Under every fact-checking system, DecModellarge yields lower Balanced Accuracy than both the 'Not Decompose' baseline and GPT-4 decomposition—e.g., SummaC-Conv 55.5% vs 62.1%; MiniCheck-FT5 72.4% vs 74.7%. The authors describe this as 'a slight decrease' and attribute it to aggregation, but the direction of the effect is uniformly negative. This directly contradicts the stated practical benefit.
  3. [§3.1.3, Eq. (3)] SEMANTIC ENTROPY as defined is not a bounded quality measure. With no semantic overlap, each atomic claim forms its own cluster and se = log n, so the metric increases monotonically with the number of atomic claims. A decomposition into many trivial or fragmented claims would receive a high score without being high quality. The paper does not control for atomicity/granularity when applying this metric, so high SEMANTIC ENTROPY cannot be interpreted as high decomposition quality.
  4. [§5.2, Table 3] Claim2Atom includes DecData test pairs generated from the same decomposition trees and synthetic reverse-checking process used to train DecMetrics. Consequently, Table 3 evaluates DecModel on the very distribution the reward model was optimized for. The high CORRECTNESS scores of the T5 models are not evidence of generalizable decomposition quality. An evaluation on independently annotated examples (e.g., human-curated FActScore/WICE samples) is needed.
minor comments (3)
  1. [Algorithm 1] In Algorithm 1, 'DecomposeRecursively' appends to 'result' but the initial call passes 'atomic_claims' without returning or updating it, so the pseudocode does not actually output the list. Also, the 'result' parameter is not used consistently in the recursive calls.
  2. [Table 1] The row labeled 'DecMetrics' is a fine-tuned DeBERTa-v3-large; please state the fine-tuning hyperparameters and note whether the same model is used for all three metrics. Also, the model name 'nli-MiniLM2-L6-H768' appears to be a typo.
  3. [§6.2/Table 4] The statement 'FactCheck-GPT datasets inherently comprise atomic facts' needs clarification, since the 'Not Decompose' baseline in Table 4 appears to apply fact-checking to full claims. Define exactly what 'Not Decompose' means for datasets that are already atomic.

Circularity Check

2 steps flagged · score 6.0 of 10

DecMetrics' core validation loop is closed: its metrics are trained and evaluated on LLM reverse-check labels, and DecModel's reward and benchmark reuse the same label source.

  1. fitted input called prediction [Section 3.2 Step 4, Section 3.2.2, Section 3.4, Table 1]
    "This process involves a reverse check using a LLM, guided by the atomic claim checking prompt detailed in Appendix Figure 7. This process verifies that the decomposed atomic claims adhere to our criteria. ... With these synthetic training data, we fine-tune DeBERTa-v3-large ... for DecMetrics with Localized Contrastive Estimation loss. ... We assess the effectiveness of automated evaluation models on both the DecMetrics test dataset and established NLI datasets ... DecMetrics 99.14%"

    The supported/unsupported labels for COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY are produced by the reverse-check LLM, which is itself the arbiter of 'complete/correct/independent.' DecMetrics models are fitted to those labels and then 'evaluated' on a held-out split of the same synthetic process. The near-perfect accuracy in Table 1 therefore measures how well the fine-tuned model reproduces the reverse-check LLM's judgments on the same distribution, not whether those judgments correspond to human decomposition quality. No human annotation or external correlation is provided; the SNLI/MultiNLI rows show the trained metrics are worse than off-the-shelf NLI models, so the external signal does not break the loop.

  2. fitted input called prediction [Section 4.1, Section 5.2, Table 3]
    "R(c, ac) = α · cp(c, ac) + β · cr(c, ac) + γ · se(ac) ... From the structured decomposition trees and their associated subtrees, we construct the dataset DecData, consisting of pairs in the form of (claim, atomic claims)."

    DecModel is optimized with PPO to maximize exactly the three DecMetrics scores that were trained on the synthetic reverse-check labels, and it is then evaluated on Claim2Atom's DecData, which is built from the same decomposition trees and the same reverse-check filtering. The reported COMP./CORR./SEM scores in Table 3 thus show the model maximizing its own reward and being measured with that same reward's label source; the loop is closed and cannot independently establish a 'standardized' high-quality decomposition model.

full rationale

The paper's central claim is that DecMetrics automatically measure decomposition quality. That claim rests entirely on the synthetic pipeline of Section 3.2: Qwen3-32B decomposes Wikipedia summaries (Step 3) and an LLM reverse-check (Step 4, Appendix Figure 7) labels subtrees as complete/correct/independent. The fine-tuned DecMetrics models are trained on these labels and then validated on a held-out split of the same pipeline, so Table 1's high accuracy is self-consistency with the label generator rather than external evidence. The PPO reward in Section 4.1 is a weighted sum of these same metrics, and the Claim2Atom evaluation (including DecData) is constructed from the same decomposition trees and reverse-check process; optimizing and then measuring with the same unvalidated objective closes the loop. The paper also reports no human annotation, no inter-annotator agreement, and no correlation with human quality judgments, and its Limitations only concedes generalizability rather than label validity. Table 4 additionally shows DecModel_large underperforming GPT-4 and no-decomposition baselines on all downstream fact-checking systems, so the practical benefit claim is not independently supported (a correctness risk, not itself a circularity). There is no reliance on the author's own prior work, so self-citation patterns are absent; the circularity is of the fitted-input-called-prediction type. Score 6 reflects that the central metric's validation and the model's reward/evaluation loop reduce to the same synthetic label source, while some independent ingredients (NLI architectures, public NLI datasets, downstream aggregation) remain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution rests on four domain assumptions and four hand-chosen parameters. The most serious is the first, because it means the metric labels and the reward signal are produced by the same kind of model the paper wants to evaluate. The alpha/beta/gamma weights are set to 1 with no sensitivity analysis, and the NLI decision threshold is never specified.

free parameters (4)
  • alpha (COMPLETENESS reward weight) = 1 (default)
    Section 4.2: 'we set alpha, beta, and gamma to 1 by default and performed ablations by zeroing out individual coefficients.' No grid search or sensitivity analysis is reported.
  • beta (CORRECTNESS reward weight) = 1 (default)
    Same as alpha.
  • gamma (SEMANTIC ENTROPY reward weight) = 1 (default)
    Same as alpha.
  • NLI entailment threshold for supported/unsupported = not specified
    The metrics and the semantic entropy clustering depend on a supported/unsupported decision boundary of an NLI model, but the threshold is never stated in the paper.
assumptions (4)
  • domain assumption The synthetic dataset generated by LLM decomposition and reverse checking provides valid ground truth for decomposition quality.
    Section 3.2: Wikipedia summaries are decomposed by Qwen 3-32B and then verified by an LLM reverse-check prompt (Figure 7). The entire supervision signal for both the metrics and the decomposition model depends on these machine-generated labels being correct. No human annotation or agreement measure is provided.
  • domain assumption NLI entailment scores reliably measure semantic relations between claims and atomic claims for completeness and correctness.
    Sections 3.1.1 and 3.1.2 define completeness and correctness through NLI probabilities on 'supported' labels. This assumes that textual entailment captures the full notion of 'all necessary aspects' and 'no fabrication' in decomposition.
  • domain assumption The aggregation rule that a claim is supported only if all atomic claims are supported is appropriate for fact-checking.
    Section 6.2: the downstream evaluation uses an all-atomic-claims-must-pass aggregation. The paper itself attributes the performance drop to this rule, so this assumption is both load-bearing and questionable.
  • domain assumption Wikipedia summary texts are representative of the distribution of real claims in fact-checking.
    Section 3.2 samples 200 entities from Wikipedia; the authors acknowledge in Section 8 that entity selection may introduce bias. The benchmark transfer to FActScore/WICE depends on this representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DecMetrics: Structured Claim Decomposition Scoring for Factually Consistent LLM Outputs." pith.science (2026). https://pith.science/paper/DLIGNWWW

@misc{pith2026250904483,
  author       = {Pith},
  title        = {Pith review of: DecMetrics: Structured Claim Decomposition Scoring for Factually Consistent LLM Outputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLIGNWWW}},
  note         = {Machine review of arXiv:2509.04483}
}
read the original abstract

Claim decomposition plays a crucial role in the fact-checking process by breaking down complex claims into simpler atomic components and identifying their unfactual elements. Despite its importance, current research primarily focuses on generative methods for decomposition, with insufficient emphasis on evaluating the quality of these decomposed atomic claims. To bridge this gap, we introduce \textbf{DecMetrics}, which comprises three new metrics: \texttt{COMPLETENESS}, \texttt{CORRECTNESS}, and \texttt{SEMANTIC ENTROPY}, designed to automatically assess the quality of claims produced by decomposition models. Utilizing these metrics, we develop a lightweight claim decomposition model, optimizing its performance through the integration of these metrics as a reward function. Through automatic evaluation, our approach aims to set a benchmark for claim decomposition, enhancing both the reliability and effectiveness of fact-checking systems.

Figures

Figures reproduced from arXiv: 2509.04483 by the authors.

Figure 1
Figure 1. Characteristics of Claim Decomposition: High Completeness, High Correctness, and High Semantic Entropy. The process involves breaking down long-form text into atomic claims. High-quality decompositions are indicated by green, red, and purple boxes, whereas orange boxes denote claims that may be deficient in essential information, introduce misinformation, or ex￾hibit redundant semantic overlap. and "atomic propositi… view at source ↗
Figure 2
Figure 2. Synthetic Data Generation Pro￾cess. The sequential process includes topic sampling from Wikipedia, extracting textual summaries as original claims, decomposing claims into atomic claims while adjusting gran￾ularity, and constructing a decomposition tree that pairs claims with atomic claims. c ac1 ac1.1 ac1.2 ac2 ac3 The Mitrokhin Commission was an Italian parliamentary commission set up in 2002 to investigate allege… view at source ↗
Figure 4
Figure 4. Illustrations for different concepts: (a) Exam [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Samples for unsupported data in reverse checking process: claim and atomic claims. systems as these models inherently infer context￾dependent information, particularly when retrieval stages provide context alignment. While decontex￾tualization can improve readability b…
Figure 6
Figure 6. Figure 6: Claim Decomposition Prompt with Chain of Thought [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Atomic Claims Verifying Prompt with Chain of Thought [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Atomic Claims Verifying Prompt with Chain of Thought (Continued) [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    If yes, return it as is

    Check if the Claim is simple. If yes, return it as is

    work page

  2. [2]

    If complex, identify key components and break into distinct Atomic Claims

    work page

  3. [3]

    arXiv preprint

    Long-form factuality in large language models. arXiv preprint. ArXiv:2403.18802 [cs]. Adina Williams, Nikita Nangia, and Samuel Bowman

    arXiv

  4. [4]

    Preserve Named Entities and replace pronouns with full names

    work page

  5. [5]

    Example Walkthrough: Claim: Ash is known for her comedic timing and her ability to play a wide range of characters, from quirky and offbeat to more serious and dramatic roles

    Verify each fact is clear, distinct, and self-contained. Example Walkthrough: Claim: Ash is known for her comedic timing and her ability to play a wide range of characters, from quirky and offbeat to more serious and dramatic roles. Steps:

    work page

  6. [7]

    Ensure Atomic Claims cover the full meaning without redundancy

    work page

  7. [10]

    • Ash is known for her ability to play a wide range of characters

    Break into: • Ash is known for her comedic timing. • Ash is known for her ability to play a wide range of characters. • The range includes quirky and offbeat roles. • The range includes more serious and dramatic roles

    work page

  8. [12]

    Named Entities (Ash) preserved

    work page

Show all 29 references
  1. [13]

    Atomic Claims: <answer> - Ash is known for her comedic timing

    Atomic Claims are clear and distinct. Atomic Claims: <answer> - Ash is known for her comedic timing. - Ash is known for her ability to play a wide range of characters. - The range includes quirky and offbeat roles. - The range includes more serious and dramatic roles. </answer...

  2. [14]

    • Competitors came from 11 nations

    Break into: • 38 artistic roller skating competitors participated. • Competitors came from 11 nations

  3. [15]

    Atomic Claims cover all aspects

  4. [16]

    Named Entities preserved

  5. [17]

    Atomic Claims: <answer> - 38 artistic roller skating competitors participated

    Atomic Claims are clear and distinct. Atomic Claims: <answer> - 38 artistic roller skating competitors participated. - Competitors came from 11 nations. </answer> Claim: The artistic roller skating tournaments took place between 22 and 23 July. Steps:

  6. [18]

    Atomic Claims: <answer> - The artistic roller skating tournaments took place between 22 and 23 July

    Return as is. Atomic Claims: <answer> - The artistic roller skating tournaments took place between 22 and 23 July. </answer> Claim: [Claim] Atomic Claims Verifying Prompt Verify the Atomic Claims against the Original Claim for completeness, correctness, and inde- pendence. Steps:

  7. [19]

    Aggregate the Atomic Claims to ensure they fully represent the meaning of the Original Claim

  8. [20]

    Check that each Atomic Claim is factually correct and contains no information absent from the Original Claim

  9. [21]

    Identify any semantic overlap between Atomic Claims, ensuring each is distinct and independent. Example Walkthrough: Original Claim: Ash is known for her comedic timing and her ability to play a wide range of characters, from quirky and offbeat to more serious and dramatic rol...

  10. [22]

    The Atomic Claims collectively represent the complete Original Claim

  11. [23]

    Each Atomic Claim is factually accurate and sourced directly from the Original Claim

  12. [24]

    There is no semantic overlap; each Atomic Claim is distinct. Conclusion: <answer> - complete - correct - independent </answer> Figure 7: Atomic Claims Verifying Prompt with Chain of Thought Atomic Claims Verifying Prompt (Continued) Original Claim: 38 artistic roller skating c...

  13. [25]

    The Atomic Claims do not fully represent the Original Claim

  14. [26]

    An Atomic Claim is factually inaccurate, diverging from the Original Claim

  15. [27]

    Conclusion: <answer> - not complete - not correct - independent </answer> Original Claim: The artistic roller skating tournaments took place between 22 and 23 July

    There is no semantic overlap; each Atomic Claim is distinct. Conclusion: <answer> - not complete - not correct - independent </answer> Original Claim: The artistic roller skating tournaments took place between 22 and 23 July. Atomic Claims: • The artistic roller skating tourna...

  16. [28]

    The Atomic Claims represent the complete Original Claim

  17. [29]

    Each Atomic Claim is factually consistent with the Original Claim

  18. [30]

    There is semantic overlap. Conclusion: <answer> - complete - correct - not independent </answer> Original Claim: [Original Claim] Atomic Claims: [Atomic Claims] Analyze and conclude whether the Atomic Claims meet the criteria or require adjustments. Figure 8: Atomic Claims Ver...

  19. [2018]

    he” with “Lanny Flaherty

    A Broad-Coverage Challenge Corpus for Sen- tence Understanding through Inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New...

    2018 arXiv

  20. [2020]

    In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online

    BART: Denoising Sequence-to-Sequence Pre- training for Natural Language Generation, Transla- tion, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics....

    2024 arXiv

  21. [2024]

    In Proceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23, pages 53728– 53741, Red Hook, NY , USA

    Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23, pages 53728– 53741, Red Hook, NY , USA. Curran Associates Inc. Colin Raffel, Noam Shazeer, ...

    2020 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.