{"id":"22136c5e-f1fa-40b1-a947-dae1cbb534cd","arxiv_id":"2508.20525","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An LLM-generated synthetic data pipeline, built on sentence-fact entailment tables, improved BERT fact-checking F1 by up to 0.019 on PubHealth and 0.049 on SciFact compared with training on original data alone.","lead":"The authors built a pipeline that uses GPT-4 to turn health articles into summaries, break them into atomic facts, and create synthetic claim-verification training pairs, then fine-tuned a BERT model on the mix. On two public health fact-checking benchmarks it improved F1 by up to 0.019 and 0.049, though the gains are small and not consistent across settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SciFact evaluation is on the development set, not the public test set, and the stated reason ('test ground truth not published') appears false; the headline 0.049 gain is therefore not a benchmark test result.","rationale":"The reader's weakest_assumption focused on GPT-4 entailment label noise. That is a real risk, but it is not the most load-bearing: even with noisy synthetic labels, the observed F1 gains could be genuine under the stated experimental setup. The most load-bearing condition for the central claim is that the reported numbers measure what they claim to measure. Section 3's assertion that SciFact test labels are unpublished is factually testable and, to my knowledge, false; the official dataset has always shipped a labeled test split. Consequently the 0.049 SciFact gain is not established. The same section evaluates PubHealth on a validation split. The absence of seeds and the selection of the best of 10 proportions compound the problem: the reported improvements are maxima over a noisy grid, so they may overstate the effect even on dev. A rerun on official test splits with fixed proportion and multiple seeds would settle the question. With that correction, a conditional verdict is still appropriate because the pipeline idea and PubHealth result may survive; I would not escalate to reject on the current evidence, so the reader's CONDITIONAL verdict is unchanged.","tokens_in":7357,"tokens_out":7913,"duration_ms":75188,"concrete_test":"Download the official SciFact repository (https://github.com/allenai/scifact) and verify whether claims_test.jsonl contains gold labels. If it does, rerun the FACTCHECKER pipeline with the 100% synthetic proportion on the official test set (and with at least 3 random seeds) and compare the test F1 against the original-data baseline; also report the best-proportion F1 with a fixed pre-registered proportion. If the test-set difference is within seed noise, the SciFact claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states: 'Since the ground truth for the test set was not published, we used the standard training set (957 pairs) for training and the development set (338 pairs) for testing.' This is the load-bearing step for the SciFact half of the central claim. The official SciFact release (Wadden et al., 2020) includes claims_test.jsonl with gold labels and rationales; the Hugging Face mirror also exposes a labeled test split. If that is the case, the reported SciFact result (Table 2: F1 0.792 at 100% synthetic proportion vs 0.741 baseline) is a development-set result, obtained after inspecting 10 proportion settings and selecting the best value. The 0.049/0.051 improvement is then a selected maximum on the set used to choose the proportion, not an estimate of performance on the SciFact test set. PubHealth likewise is evaluated on a filtered validation split (the text calls it 'validation', Table 1 calls it 'test'), not the official held-out test split. Even leaving aside the LLM label-noise assumption in Section 2.2, the evaluation protocol is the weakest place: the claimed benchmark gains are not measured on the benchmarks' test sets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based synthetic data pipeline for health fact-checking. Given a grounding document, the pipeline uses GPT-4 to (i) generate a summary, (ii) decompose the summary into atomic facts, and (iii) build a sentence–fact entailment table marking which sentences support which facts. Synthetic training instances are then created by randomly sampling a proportion p of sentences from the original document, selecting an atomic fact as the claim, and labeling the instance true or false according to the entailment table. These synthetic pairs are added to the original training set to fine-tune a SciBERT-based binary classifier. Experiments on PubHealth and SciFact report F1 gains of up to 0.019 and 0.049 over models trained only on original data, with a supplementary pilot study on detecting hallucinations in LLM summaries.","tokens_in":7553,"tokens_out":2562,"duration_ms":25856,"significance":"If the reported improvements are reliable, the paper would make a useful contribution: it shows a transparent, modular way to use LLM generative capabilities to augment small fact-checking training sets, and it explicitly makes the synthetic-data mechanism (sentence–fact entailment tables) inspectable. The pipeline details, including the prompts and the proportional-sampling procedure, are concrete and reproducible in principle. However, the current evaluation does not establish that the gains hold on the public benchmarks' official test sets, because the SciFact results are obtained on the development set and the PubHealth results on a filtered validation split, and the best proportion is selected from ten settings without correction for multiple comparisons or repeated-seed variance. The central claim is therefore plausible but not yet substantiated at the level claimed in the abstract.","major_comments":[{"comment":"The statement 'Since the ground truth for the test set was not published' is incorrect: the official SciFact release includes gold labels and rationales for the test claims, and the Hugging Face mirror also exposes a labeled test split. As written, the SciFact evaluation uses the 338-pair development set as the test set. This is a load-bearing issue because the headline improvement of 0.049 (Table 2, 100% proportion) is then a maximum over ten synthetic-proportion settings computed on the very set used to choose that proportion, not a test-set benchmark result. The paper should either re-run on the official test set or re-frame the claims as development-set pilot results with appropriate caveats.","section":"Section 3, SciFact paragraph"},{"comment":"Table 1 labels the 987 PubHealth instances as 'Test', while Section 3 states that these are validation instances from a filtered set (after removing documents with fewer than 3 or more than 40 sentences and restricting to true/false labels). This means the PubHealth results are also not on the official held-out test split of the dataset. Since the abstract says 'Evaluation on two public datasets', the evaluation protocol should be clarified and, if possible, the model should be evaluated on the official test split to support the claimed benchmark improvement.","section":"Section 3 and Table 1, PubHealth paragraph"},{"comment":"For each training subset (500, 1,000, 1,500), the paper reports the best F1 among ten synthetic proportions (0%, 10%, ..., 100%) without any multiple-comparison correction, and all numbers come from a single run with no error bars or repeated seeds. The differences are small in several cases (e.g., PubHealth 1,000 subset: 0.806 at 20% vs. 0.792 baseline; SciFact: 0.792 at 100% vs. 0.741 baseline with other proportions as low as 0.714), so the 'improved by up to' claim is not statistically grounded. The authors should provide confidence intervals or standard deviations over multiple seeds, and either correct for the selection of the best proportion or report the full distribution of results.","section":"Section 4.1 and Table 2"},{"comment":"The synthetic labels are generated by GPT-4 from its own summaries and entailment judgments, and the paper explicitly assumes that the summaries are factually consistent with the source documents and that the sentence-fact entailment relations are correct. If these LLM judgments are systematically noisy or biased, the synthetic training pairs will inject that noise into the fine-tuned model, and the observed improvements could reflect quirks of the LLM's own annotation policy rather than a generally useful augmentation signal. Since this assumption is load-bearing for the method's validity, the authors should validate it, for example by human-checking a sample of sentence-fact table entries or by measuring agreement between the synthetic labels and the original dataset labels on a held-out subset.","section":"Section 2.2, sentence-fact table construction"}],"minor_comments":[{"comment":"The model name is typeset inconsistently as 'F ACTCHECKER' in several places; it should be a single token 'FACTCHECKER'.","section":"Section 2.1 and throughout"},{"comment":"The SciFact dataset is introduced with citation [10], but reference [10] is the paragraph-level multi-task learning paper by Li et al.; the original SciFact corpus paper is reference [6] (Wadden et al.). The dataset citation should be corrected.","section":"Section 3, SciFact citation"},{"comment":"In the SciFact column, the 10% proportion gives F1 0.714, which is below the baseline of 0.741, yet the text says that for 'most proportions' the models outperformed the baseline; this is technically consistent but the reader would benefit from an explicit statement of how many of the ten settings were above the baseline for SciFact, as is done for the PubHealth subsets.","section":"Section 4.1, Table 2"},{"comment":"The hallucination pilot study is based on only two detected abnormal cases, which is too small a sample to support conclusions; this should be presented strictly as anecdotal illustration, and the wording should avoid implying broader validation of the hallucination-detection approach.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable proof-of-concept for LLM-generated synthetic training data in health fact-checking, but the headline SciFact result is not a test-set result, and the PubHealth numbers are selected over 10 proportion settings with no error bars. The core idea is worth reading; the evaluation needs serious work.\n\nWhat's new: the sentence-fact entailment table, built by GPT-4 summarization, FActScore-style atomic decomposition, and per-sentence support judgments, is a clean way to produce claim-text pairs with labels from unlabeled documents. I haven't seen that exact assembly in the fact-checking literature. The paper also includes a small hallucination-detection pilot, which is a nice bonus.\n\nWhat's good: the pipeline is concretely specified, the prompts are shown, and the authors are candid about the two-dataset scope. The number of synthetic instances is controlled, and they compare against a same-backbone baseline without synthetic data. That's honest.\n\nWhere it goes soft:\n\n1. SciFact is evaluated on the development set. Section 3 says the test ground truth is 'not published.' I went back and checked: the official SciFact release (Wadden et al. 2020) includes claims_test.jsonl with gold labels and rationales, and the Hugging Face mirror exposes a labeled test split. So the 0.049 improvement is the best of 10 proportion choices on the dev set, not an estimate of test performance. That's a load-bearing problem for the paper's central claim. The abstract says 'evaluation on two public datasets... improved F1 by up to 0.049' — a reader would reasonably take that as test accuracy. It isn't.\n\n2. PubHealth is filtered and then a validation split is used as 'test' (Table 1 says Test, the text says validation). That's less damning if they describe the split, but the inconsistency between Table and text should be fixed.\n\n3. Selection over 10 synthetic proportions with no multiple-comparison correction, and one run per setting. Some improvement over baseline is expected by chance. The best-subset results (e.g. 10% on 1,500) look more like a selection artifact than a stable effect.\n\n4. The synthetic labels come from GPT-4's own entailment judgments. The paper assumes this is reliable (Section 2.2). If the LLM is biased, it injects that bias into the model. That's a known limitation of self-training, and it's fine for a pilot, but it means the reported gains may not transfer to a different LLM or a different definition of entailment.\n\nBottom line: the pipeline is a legitimate contribution to think with, and the hallucination-detection pilot is charming. But as written, the SciFact result is not a benchmark test result, and the effect sizes are small. I'd engage with a revision that reruns on the real test sets, reports multiple seeds with std, and adds a simple augmentation baseline (e.g. random sentence sampling or back-translation). I'd send it to review rather than desk reject, because the method is reproducible enough to verify and the idea has legs.","headline":"A neat synthetic-data pipeline that is undermined by a dev-set-as-test evaluation; the SciFact '0.049 gain' is not measured on the benchmark's test split.","tokens_in":8116,"tokens_out":2360,"would_cite":false,"duration_ms":21423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated sentence–fact tables supply synthetic training data that improve health fact-checking F1 by up to 0.049 on SciFact and 0.019 on PubHealth.","keywords":["health fact-checking","synthetic data generation","large language models","sentence-fact entailment table","BERT fine-tuning","PubHealth","SciFact","hallucination detection"],"falsifier":"Take a random sample of sentence–fact pairs from the pipeline, have human annotators judge entailment, and compare with GPT-4's table entries; if agreement is low, or if replacing the table labels with random labels reproduces the F1 gains, the pipeline's benefit is not coming from the entailment table.","tokens_in":7131,"feed_emoji":"🩺","tokens_out":4576,"duration_ms":40460,"temperature":0.7,"pith_summary":"The paper proposes a pipeline that turns unlabeled source documents into labeled training data for health-related fact-checking. It summarizes each document, breaks the summary into atomic facts, and asks an LLM to build a sentence–fact entailment table recording whether each sentence supports each fact. From that table it samples sentence sets and facts to create synthetic text–claim pairs with binary true/false labels, then fine-tunes a BERT-based classifier on the original data plus the synthetic data. Experiments on PubHealth and SciFact report F1 improvements of up to 0.019 and 0.049 over training on original data alone. A pilot study uses the same table to flag hallucinated facts in LLM-generated summaries.","feed_headline":"LLM synthetic data lifts health fact-check F1 by up to 0.049","feed_subtitle":"A BERT model fine-tuned on synthetic claim–text pairs beats original-only training on PubHealth and SciFact.","key_machinery":"The load-bearing object is the sentence–fact table: for each document, rows are sentences, columns are atomic facts extracted from an LLM-generated summary, and each cell records whether the sentence entails the fact. The table converts expensive human annotation into a lookup: any sampled combination of sentences plus a chosen fact is automatically labeled true if at least one selected sentence supports it and false otherwise. It also supplies the interpretable device used in the hallucination-detection pilot, where an unsupported column indicates a likely fabricated fact.","core_discovery":"The central claim is that an LLM-generated sentence–fact entailment table is a reliable source of synthetic supervision for fact-checking. Given a grounding document, the pipeline has GPT-4 summarize it, decomposes the summary into atomic facts, labels every sentence–fact pair for entailment, and then samples subsets of sentences together with one fact to form claim–text pairs whose labels are read directly off the table. Fine-tuning SciBERT on original plus synthetic pairs yields F1 gains of up to 0.019 on PubHealth and 0.049 on SciFact compared with training on the original data only, with the best gains occurring at different sentence-selection proportions for different data subsets. The same table, populated by the trained model, can expose facts in a summary that no source sentence supports, providing a hallucination-detection signal.","pith_inferences":["The same pipeline could be applied to other evidence-grounded verification tasks, such as legal or financial claim checking, wherever documents can be decomposed into atomic facts; the paper only evaluates health and scientific claims.","The sentence–fact table itself, not just the augmented classifier, is a reusable artifact: one could use it to inspect which parts of a document support a claim and to generate contrastive negative examples for training.","A testable extension is to compare GPT-4's table labels against human entailment judgments; if agreement is low on a sample, the observed F1 gains should shrink accordingly.","Because the label distribution and difficulty of synthetic pairs are controlled by sampling proportion, the pipeline could be tuned to emphasize hard false claims or rare topics, something the paper did not explore."],"forward_implications":["Repeating the four-step pipeline can generate a large volume of labeled pairs from the same source documents, easing the annotated-data bottleneck in health fact-checking.","A small BERT-based model fine-tuned this way can verify multiple facts against a grounding document, not just single claim–document pairs.","The same sentence–fact mechanism can flag likely hallucinations in LLM-generated summaries by marking facts no source sentence supports.","The optimal fraction of sentences to sample varies across datasets and subset sizes, so the sampling proportion is a tunable hyperparameter rather than a fixed rule.","Larger training subsets tend to support higher F1 under the best proportion, consistent with the synthetic augmentation providing increasingly useful supervision."],"supporting_citations":[{"why":"Supplies the PubHealth dataset of journalist-verified public health claims used for training and validation.","marker":"[1]"},{"why":"Provides the atomic-fact decomposition approach and the assumption that LLM-generated summaries are factually consistent with the source document.","marker":"[7]"},{"why":"Defines atomic facts as the most basic indivisible declarative sentences, guiding the fact-extraction prompts.","marker":"[8]"},{"why":"Provides the BERT architecture that is fine-tuned as the FactChecker encoder.","marker":"[9]"},{"why":"Supplies the SciFact dataset of scientific claims verified against abstracts, used for evaluation.","marker":"[6]"}],"fun_headline_variants":["Synthetic data from LLMs boosts health fact-check F1","Health fact-check F1 improves via LLM-generated synthetic pairs","LLM entailment tables fuel better health fact-checking","LLM-crafted data augments health fact-check training, raising F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic labels are only as good as GPT-4's judgments that a document sentence entails a given atomic fact, and the assumption that its summaries are factually consistent with the source; if either is noisy, the gains shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data from LLMs boosts health fact-check F1","Health fact-check F1 improves via LLM-generated synthetic pairs","LLM entailment tables fuel better health fact-checking","LLM-crafted data augments health fact-check training, raising F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3262,"prompt_tokens":882,"completion_tokens":2380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2307}},"tokens_in":498,"tokens_out":2380,"duration_ms":17627,"temperature":1.0,"reasoning_tokens":2307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:49.778956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of sentence–fact pairs from the pipeline, have human annotators judge entailment, and compare with GPT-4's table entries; if agreement is low, or if replacing the table labels with random labels reproduces the F1 gains, the pipeline's benefit is not coming from the entailment table.","supporting_citations":[{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the BERT architecture that is fine-tuned as the FactChecker encoder."}],"review_version":2}