{"id":"07fd8d22-5099-4e9a-8869-611f7699cec9","arxiv_id":"2509.04483","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DecMetrics scores claim decompositions for completeness, correctness, and semantic entropy, and uses them as rewards to train a small T5 model, but the evaluation is circular and it does not lift downstream fact-checking accuracy.","lead":"This paper proposes DecMetrics, three automatic scores for judging whether an LLM's claim decomposition is complete, correct, and non-redundant. A small model trained with these scores as rewards matches larger LLMs on the authors' benchmark, but the validation loop is synthetic and downstream fact-checking does not improve.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-generated reverse-check labels are never validated against human judgment; if they are wrong, the DecMetrics scorers, the PPO reward, and all Claim2Atom evaluations optimize an arbitrary target.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing condition: the synthetic data generation process must produce ground-truth labels that correspond to human judgments. I agree. The paper's internal logic is coherent—the three metric definitions are clear, the data-construction steps are consistent, and the training pipeline follows from the definitions—but the empirical grounding is absent. No human annotation or external validation appears anywhere in the manuscript. The holdout test set in §3.2.2 is generated by the same LLM pipeline, so high accuracy on it does not establish validity. The downstream fact-checking results actually weaken the practical claim, since the trained model does not beat baselines. Given this, the current REJECT verdict is appropriate; the proposed human-validation study would be the decisive check that could change it. I therefore leave the verdict unchanged.","tokens_in":16182,"tokens_out":4713,"duration_ms":61193,"concrete_test":"Sample 100 (claim, atomic_claims) instances from the DecMetrics test split or DecData test. Have three trained annotators independently label each instance for completeness, correctness, and independence using the same rubric as the Appendix Fig. 7 prompt, plus a 5-point holistic quality score. Compute (i) pairwise Cohen's kappa among annotators and kappa between the majority human label and the LLM reverse-check label for each of the three dimensions; (ii) Spearman correlation between the fine-tuned DecMetrics scores (cp, cr, se) and mean human quality ratings. If annotator–LLM kappa is below 0.6 on any dimension, or if any metric's correlation with human quality is not significantly positive, the synthetic labels are not a valid proxy for decomposition quality and the central claim fails. If agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY measure decomposition quality—rests entirely on labels produced by the synthetic pipeline in §3.2. Step 3 uses Qwen3-32B to generate decompositions; Step 4 uses an LLM reverse-check (Appendix Fig. 7) to label subtrees as complete/correct/independent. The paper reports no human annotation, no inter-annotator agreement, and no external correlation with human quality judgments. The fine-tuned DecMetrics models, the PPO reward R = α·cp + β·cr + γ·se (§4.1), and the Claim2Atom evaluation in Table 3 are all built on these machine-generated labels. If the reverse-check LLM systematically accepts decompositions that human annotators would call incomplete, incorrect, or redundant—or rejects good ones—every downstream number is measuring the LLM's own bias rather than decomposition quality. This is not a missing nicety: Table 1's high accuracy merely shows agreement with the same pipeline that generated the labels. The downstream evidence in Table 4 also shows DecModel_large underperforming both GPT-4 and no decomposition on every fact-checking system, so the claimed practical benefit is not independently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DecMetrics, three automatic metrics (COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY) intended to evaluate the quality of claim decomposition in LLM-generated text. The metrics are implemented with NLI-based models fine-tuned on synthetic data produced by a Wikipedia-based pipeline: Qwen3-32B decomposes claims and an LLM reverse-check labels the resulting subtrees. The same metrics are used as a reward signal in a three-stage PPO pipeline to train DecModel, a lightweight T5-based decomposition model. The paper also introduces Claim2Atom, a benchmark combining FActScore, WICE, and the new DecData. Experiments report high internal accuracy for DecMetrics on its own test set, competitive scores for DecModel on Claim2Atom, and downstream fact-checking results that show a decrease relative to baselines.","tokens_in":16517,"tokens_out":5344,"duration_ms":65026,"significance":"The idea of formalizing decomposition quality into structured reward components and training a compact model with them is interesting and could be useful if the metrics were valid. The paper releases code and data, and the three-stage SFT/reward/PPO recipe is clearly described. However, the central claim is not established: the metrics are validated only against the same synthetic label generator that created them, and the main downstream experiment shows that DecModel underperforms both GPT-4 decomposition and no decomposition. The contribution, as presented, is therefore not supported.","major_comments":[{"comment":"The supported/unsupported labels for all three metrics are produced by the pipeline in Steps 3–4: Qwen3-32B decompositions and an LLM reverse-check (Appendix Fig. 7). The fine-tuned DecMetrics scorers are trained and tested on synthetic examples derived from this same pipeline (§3.2.2). Table 1's 99.14% accuracy therefore measures agreement with the label generator, not with any independent notion of decomposition quality. No human annotation, inter-annotator agreement, or correlation with existing quality judgments is reported. Since the PPO reward in §4.1 and the Claim2Atom evaluation in Table 3 inherit these labels, the central claim that the metrics assess quality is unsupported.","section":"§3.2, Table 1"},{"comment":"The downstream experiment fails to support the abstract's claim that the approach enhances fact-checking. Under every fact-checking system, DecModellarge yields lower Balanced Accuracy than both the 'Not Decompose' baseline and GPT-4 decomposition—e.g., SummaC-Conv 55.5% vs 62.1%; MiniCheck-FT5 72.4% vs 74.7%. The authors describe this as 'a slight decrease' and attribute it to aggregation, but the direction of the effect is uniformly negative. This directly contradicts the stated practical benefit.","section":"§6.2, Table 4"},{"comment":"SEMANTIC ENTROPY as defined is not a bounded quality measure. With no semantic overlap, each atomic claim forms its own cluster and se = log n, so the metric increases monotonically with the number of atomic claims. A decomposition into many trivial or fragmented claims would receive a high score without being high quality. The paper does not control for atomicity/granularity when applying this metric, so high SEMANTIC ENTROPY cannot be interpreted as high decomposition quality.","section":"§3.1.3, Eq. (3)"},{"comment":"Claim2Atom includes DecData test pairs generated from the same decomposition trees and synthetic reverse-checking process used to train DecMetrics. Consequently, Table 3 evaluates DecModel on the very distribution the reward model was optimized for. The high CORRECTNESS scores of the T5 models are not evidence of generalizable decomposition quality. An evaluation on independently annotated examples (e.g., human-curated FActScore/WICE samples) is needed.","section":"§5.2, Table 3"}],"minor_comments":[{"comment":"In Algorithm 1, 'DecomposeRecursively' appends to 'result' but the initial call passes 'atomic_claims' without returning or updating it, so the pseudocode does not actually output the list. Also, the 'result' parameter is not used consistently in the recursive calls.","section":"Algorithm 1"},{"comment":"The row labeled 'DecMetrics' is a fine-tuned DeBERTa-v3-large; please state the fine-tuning hyperparameters and note whether the same model is used for all three metrics. Also, the model name 'nli-MiniLM2-L6-H768' appears to be a typo.","section":"Table 1"},{"comment":"The statement 'FactCheck-GPT datasets inherently comprise atomic facts' needs clarification, since the 'Not Decompose' baseline in Table 4 appears to apply fact-checking to full claims. Define exactly what 'Not Decompose' means for datasets that are already atomic.","section":"§6.2/Table 4"}],"recommendation":"reject","confidential_remarks":"The paper has a single negative downstream result and no human validation of the synthetic labels. Both are load-bearing issues for the central claim. I would not be willing to accept a revision unless the authors provide a human-annotated evaluation and substantially reframe the claimed contribution, which would be a major new study rather than a standard revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the DecMetrics paper. The core idea is reasonable: decompose claim-decomposition quality into completeness, correctness, and semantic entropy, then use them as rewards for a decomposition model. That three-way decomposition is a legitimate framing, and the paper does a clean job of defining the metrics as NLI-based recall, precision, and redundancy, respectively. The authors also ship code and data, and they report a downstream fact-checking table (Table 4) where their trained DecModel underperforms both GPT-4 decomposition and no decomposition on every system. Reporting that negative result is honest, but it directly contradicts the abstract's claim that the approach enhances fact-checking reliability.\n\nThe problem is not the math; it's the validation. The labels used to train the DecMetrics scorers come from an LLM decomposition and LLM reverse-check pipeline (Section 3.2), and the evaluation in Section 3.4 uses a held-out split from that same synthetic process. There is no human annotation, no inter-annotator agreement, and no external correlation with human judgments. Table 1's high accuracy is therefore just agreement with the label generator. The fine-tuned DeBERTa scorers, the PPO reward, and the Claim2Atom benchmark are all built on that unvalidated ground truth. If Qwen3-32B's reverse-check systematically accepts decompositions humans would reject (or rejects good ones), every number in the paper measures the LLM's own bias. The paper's Limitations section (Section 8) mentions synthetic-data generalization and entity bias, but it is silent on this deeper circularity.\n\nI don't think this is a fatal idea. The three-metric decomposition is useful, and the paper's framing of why completeness and redundancy matter (e.g., the ClaimScore overlap blind spot) is well grounded in prior work like Hu et al. 2024 and Wanner et al. 2024. But the evidence as presented is not sufficient to support the claim that DecMetrics measure decomposition quality in any generalizable sense. What's missing is a small human-annotated evaluation set, even a few hundred examples, plus a correlation study with existing decomposition quality judgments. Without that, the paper is a promising framework with an unvalidated core.\n\nWould I cite it? Not in the next year, because I couldn't rely on its results. Would I take it to reading group? Maybe, as a case study in synthetic-data circularity in the fact-checking space. Would I send it to peer review? Yes, if the authors are willing to add human validation; the idea deserves a proper airing, but the version in front of me needs substantial revision or a much more cautious conclusion. I'd tell the authors to focus on the negative result and frame the paper as an open problem rather than a benchmark claim.","headline":"The metric definitions are clear and the combination is new, but the paper's evidence that they measure decomposition quality is circular and the downstream results undercut the central claim.","tokens_in":16971,"tokens_out":1697,"would_cite":false,"duration_ms":25378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DecMetrics introduces three automatic scores—coverage, correctness, and non-redundancy—for judging whether a claim decomposition is good enough to trust.","keywords":["claim decomposition","fact-checking","atomic claims","natural language inference","semantic entropy","reinforcement learning","LLM factuality","benchmark"],"falsifier":"Take a random sample of claims from Claim2Atom, have several human annotators independently judge each decomposition on the three axes, and compare their labels with DecMetrics' scores. If agreement on any axis is no better than chance, the claim that these metrics measure decomposition quality fails. A more direct check: for Correctness, construct decompositions that humans judge 50/50 on whether an atomic claim was present in the original text, and see whether the NLI scorer's supported/unsupported boundary tracks the human boundary.","tokens_in":16097,"feed_emoji":"🧩","tokens_out":10057,"duration_ms":102464,"temperature":0.7,"pith_summary":"Fact-checking systems that verify long-form AI output usually start by chopping the text into tiny atomic claims, and the quality of that chop determines the final verdict. The paper argues that this decomposition step is rarely evaluated, and a bad one can make a false text look true or a true text look false. To fix this, DecMetrics defines three automatic scores—Completeness, Correctness, and Semantic Entropy—which measure whether the atomic claims together cover the original claim, stay faithful to it, and avoid saying the same thing twice. The authors build synthetic training data from Wikipedia summaries, teach NLI models to compute the three scores, and then use those scores as a reward to train a compact decomposition model. Their result is a claim-decomposition benchmark, Claim2Atom, plus a small model that, according to the paper, produces decompositions competitive with much larger LLMs.","feed_headline":"Three scores grade claim decompositions for fact-checking","feed_subtitle":"DecMetrics scores become a reward that steers a compact model to split claims faithfully","key_machinery":"The decomposition tree is the load-bearing structure. Wikipedia summaries are split recursively until each claim is non-splittable, producing a hierarchy of claims and atomic claims. Sibling subtrees yield training pairs: dropping leaves produces 'incomplete' negatives, claims paired with foreign nodes produce 'incorrect' negatives, and overlapping sibling claims produce 'redundant' positives for the Semantic Entropy classifier. The same hierarchy supplies the composite reward R = α·Completeness + β·Correctness + γ·Semantic_Entropy, which is used to train DecModel via PPO, and the filtered subtrees form DecData, the new component of Claim2Atom.","core_discovery":"DecMetrics frames decomposition quality as three entailment-based scores. Completeness is the entailment probability of the merged atomic claims given the original claim; Correctness is the fraction of atomic claims entailed by the original claim; Semantic Entropy clusters atomic claims that mutually entail each other and reports the entropy of that clustering, so that repeated paraphrases lower the score. The paper's central claim is that these three axes capture the ways decomposition goes wrong—omitted information, fabricated facts, and redundant overlap—and that they can be learned automatically. Training labels come from a synthetic pipeline that recursively decomposes Wikipedia summari","pith_inferences":["The paper leaves implicit that its synthetic labels come from an LLM reverse check with no human annotation; a natural next test is whether human raters agree with DecMetrics on the same decompositions, since low agreement would mean the metrics are calibrated to LLM judgment rather than to what readers count as missing or invented.","Semantic Entropy, as defined, penalizes any mutual entailment between atomic claims, which could suppress legitimate elaboration; a testable refinement would weight redundancy by whether repeated content is load-bearing for verification.","The same decomposition-tree machinery could transfer to other structured text problems—for example, evaluating retrieval-augmented answers for coverage and hallucination—by substituting arbitrary source documents for Wikipedia summaries."],"forward_implications":["Fact-checking systems can now score their own decomposition step before trusting it, catching omissions, fabrications, and duplicates before they distort the final factuality score.","Because the reward is composite, the training recipe suggests that a small model can be pushed toward all three quality axes at once, offering a low-cost alternative to prompting huge LLMs.","Claim2Atom gives later work a single benchmark with three measured axes, so decomposition models can be compared on coverage and fidelity rather than on output format alone.","The metrics inherit all limits of NLI models; the paper shows off-the-shelf NLI models score poorly on its test set, which means task-specific fine-tuning is a necessary part of the pipeline.","The ablations imply the three metrics are complementary, not redundant: dropping any one component measurably reduces the score it controls."],"supporting_citations":[{"why":"Defines atomic facts and supplies the FActScore dataset that Claim2Atom reuses; establishes the decomposition-first fact-checking paradigm DecMetrics evaluates.","marker":"Min et al., 2023"},{"why":"Supplies the WiCE dataset and the entailment framing of claim completeness that DecMetrics adapts into its Completeness metric.","marker":"Kamoi et al., 2023"},{"why":"Prior decomposition-quality metric (ClaimScore) that DecMetrics positions against; the paper argues ClaimScore ignores overlap and completeness.","marker":"Wanner et al., 2024"},{"why":"Origin of semantic entropy, repurposed here as a diversity score over clusters of atomic claims.","marker":"Farquhar et al., 2024"},{"why":"Provides the PPO algorithm used to optimize DecModel with DecMetrics as reward.","marker":"Schulman et al., 2017"},{"why":"DeBERTa-v3 backbone that is fine-tuned to produce the three NLI scorers.","marker":"He et al., 2021"},{"why":"T5 architectures used for the DecModel decomposition models.","marker":"Raffel et al., 2020"},{"why":"Supplies LLM-AGGREFACT test set and MiniCheck fact-checking baselines used in downstream evaluation.","marker":"Tang et al., 2024"}],"fun_headline_variants":["Three metric scores rate claim decomposition quality","Entailment-based scores expose claim split errors","DecMetrics: three scores for fact-check decomposition","Scoring claim splits: completeness, correctness, entropy","Three entailment scores gauge claim decomposition fidelity"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the synthetic labels—produced by an LLM decomposition followed by an LLM reverse check—are correct characterisations of completeness, correctness, and independence; the paper provides no human annotation or inter-annotator agreement to establish that these labels match what human judges would call good decompositions.","fun_headline_variants_meta":{"raw":{"variants":["Three metric scores rate claim decomposition quality","Entailment-based scores expose claim split errors","DecMetrics: three scores for fact-check decomposition","Scoring claim splits: completeness, correctness, entropy","Three entailment scores gauge claim decomposition fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3095,"prompt_tokens":642,"completion_tokens":2453,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":2384}},"tokens_in":386,"tokens_out":2453,"duration_ms":21772,"temperature":1.0,"reasoning_tokens":2384,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:12:22.642083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of claims from Claim2Atom, have several human annotators independently judge each decomposition on the three axes, and compare their labels with DecMetrics' scores. If agreement on any axis is no better than chance, the claim that these metrics measure decomposition quality fails. A more direct check: for Correctness, construct decompositions that humans judge 50/50 on whether an atomic claim was present in the original text, and see whether the NLI scorer's supported/unsupported boundary tracks the human boundary.","supporting_citations":[],"review_version":1}