{"id":"f30b20ac-c71f-4201-9926-82076a888ef2","arxiv_id":"2411.09255","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DAHL is a new biomedical benchmark and pipeline that breaks long-form LLM answers into atomic facts, checks each fact with a web-augmented model, and averages the true ones into a DAHL Score.","lead":"This paper introduces DAHL, an automated benchmark that scores how often biomedical chatbot answers contain made-up facts by splitting answers into small claims and checking each one. It also reports that scaling language models beyond 7 to 8 billion parameters gives little further reduction in hallucination.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central validity claim is not established: the DAHL Score depends entirely on the unvalidated pplx-API Checker, whose 'unknown' labels silently exclude entire responses and can bias every model comparison in Table 2.","rationale":"DAHL makes a useful, public, and reproducible contribution: the 8,573-question dataset, open-source pipeline, and the paper's own limitation and ethics statements are real assets. The splitter is at least sanity-checked by a t-test on unit counts, and the 10% sampling validation is a reasonable internal consistency check, though it validates only the sampling procedure, not the checker. The automated score, however, is only as good as the pplx-API Checker, and the paper supplies no direct validation of the checker's labels while explicitly acknowledging an exclusion mechanism that can bias scores. A response-level correlation of r=0.5508 explains roughly 30% of the variance and is not evidence that the checker is unbiased at the atomic-unit level; it can coexist with systematic false or unknown errors that shift model rankings. The paper's own qualitative comparison in Section 5.3 confirms the checker is more conservative than human annotators, making selection bias a live concern rather than a purely hypothetical one. The reader's conditional verdict is therefore appropriate: the benchmark and pipeline have value, but the central validity claim needs either checker-level validation against a gold standard or a re-analysis that treats unknown labels explicitly instead of excluding them. This is an empirical validation gap, not an internal inconsistency, and it does not change the reader's recommended verdict.","tokens_in":14608,"tokens_out":3789,"duration_ms":64433,"concrete_test":"Using the released code, run the pplx-API Checker on all atomic units from the 99 human-annotated responses, treating the human labels as gold standard. Report the unknown rate, per-label precision/recall, and then recompute the DAHL Scores under three annotation policies: (i) current exclude-unknown; (ii) unknown treated as false; (iii) unknown treated as true. If the unknown rate is substantial (e.g., >5%), if the checker's false-unit recall is below ~90%, or if the model ordering in Table 2 changes under policies (ii) or (iii), then the DAHL Score cannot be considered a validated measure until the checker is calibrated and the exclusion bias is corrected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that the DAHL Score is a valid automated measure of fact-conflicting hallucination — rests entirely on factuality labels produced by the proprietary pplx-API Checker (Section 3.3, Step 2). The only evidence offered for checker reliability is the response-level Pearson correlation of 0.5508 against human scores on 99 responses (Section 5.3); the checker's atomic-unit labels are never compared against a gold standard, and no inter-annotator agreement is reported. More importantly, the paper itself states in Section 5.3 that 'If an atomic unit is labeled \"unknown,\" the response containing that atomic unit is excluded from the final DAHL Score calculation.' Thus every score in Table 2 is computed only over responses whose atomic units all received binary labels. If unknown labels occur non-uniformly across models (e.g., more often for vague or context-dependent outputs), the exclusion itself alters the score and can create or hide model differences independently of factuality. The paper never reports the exclusion rate, and a 99-response sample is too small to absorb this bias. Because the scaling conclusion (larger models hallucinate less until 7–8B) and the proposed use as preference labels both depend on these scores, this unchecked selection is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DAHL, a benchmark dataset and automated evaluation system for measuring fact-conflicting hallucination in long-form biomedical text generation. The dataset consists of 8,573 questions derived from PubMed Central papers across 29 categories. The evaluation pipeline splits model responses into atomic units using gpt-4o, checks each unit with a Perplexity API-based checker, and computes the DAHL Score as the average proportion of factual units. The authors report scores for gpt-4o and seven open-source models, analyze scaling behavior within model families, examine temperature effects, and validate the automated scores against human annotation on 99 responses, obtaining a Pearson correlation of 0.5508. The central claim is that DAHL provides a valid automated alternative to human-annotated preference labels for long-form hallucination evaluation.","tokens_in":14869,"tokens_out":2357,"duration_ms":24827,"significance":"If the validity of the automated score were established, the paper would provide a practically useful benchmark and pipeline for biomedical hallucination evaluation, with genuine strengths: the dataset and code are publicly released, the atomic-unit decomposition is motivated by a real limitation of binary response-level judgments, and the framework is designed to be extendable to other domains. The scaling observations and temperature analysis are potentially informative empirical contributions. However, the significance is conditional on validation of the proprietary checker and on ruling out selection bias from excluded responses; as reported, the evidence for the central validity claim is moderate at best.","major_comments":[{"comment":"The factuality checker (pplx-API) is the sole source of truth for all atomic-unit labels, yet the paper never validates the checker's labels against a gold standard. The only evidence offered in §5.3 is a response-level Pearson correlation of 0.5508 on 99 responses; no atomic-level agreement, precision, recall, or confusion matrix is reported. The authors should report per-unit agreement between the checker and human labels, ideally with Cohen's kappa, and characterize the error modes (e.g., false versus unknown).","section":"§3.3 Step 2 and §5.3"},{"comment":"The exclusion of responses containing atomic units labeled 'unknown' is a load-bearing selection step, but the paper reports no exclusion counts or rates. If unknown labels occur more frequently for some models or response styles, every DAHL Score in Table 2 is computed over a different, non-random subset of responses, and the model rankings could be driven by the selection rather than by factuality. The authors should report the number and fraction of excluded responses per model, and provide a sensitivity analysis (for example, treating unknown as false, or as its own category, or imputing scores from the remaining units).","section":"§5.3"},{"comment":"The claim that 'beyond a model size of 7 to 8 billion parameters, further scaling does not significantly improve factual accuracy' is not supported by any statistical test. The differences among Llama-3.1 8B vs 70B (0.8627 vs 0.8733) and Qwen-2 7B vs 72B (0.8870 vs 0.8997) are small relative to the 0.55 correlation observed in human validation, so they may be within noise. The authors should provide per-question paired tests, bootstrapped confidence intervals, or an explicit statistical comparison of score differences across model sizes.","section":"§5.1, Table 2"},{"comment":"The human validation uses only 99 responses and two annotators, but no inter-annotator agreement is reported. Without a measure such as Cohen's kappa or the correlation between the two annotators' response-level scores, it is unclear how much of the 0.5508 correlation is limited by checker error, splitter disagreement, or human annotation noise. This is essential for interpreting the validity claim.","section":"§5.3"}],"minor_comments":[{"comment":"There is a typo: 'primarliy' should be 'primarily'.","section":"§3.2.3"},{"comment":"The table lists the gpt-4o parameter size as '?', which is acceptable for a proprietary model, but the inconsistent capitalization of 'Mpt' versus 'MPT' in the same table and text should be cleaned up.","section":"Table 2"},{"comment":"The caption refers to 'Qwen-2-8b', but Table 2 lists Qwen-2-7B; the discrepancy should be corrected.","section":"Figure 5"},{"comment":"The paper says that noncommittal responses and redundant sentences are removed 'with regular expression', but the pipeline description in §3.3 Step 0 only mentions removing the prompt, deduplicating sentences, and removing incomplete sentences. The implementation of the noncommittal-response removal should be specified.","section":"§3.1"},{"comment":"Several references have formatting errors (e.g., 'V ogeti' for the Llama 3 author list); the reference list should be proofread.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is incremental over FactScore but potentially useful for the biomedical domain. The decisive issue is whether the authors can strengthen validation of the checker and quantify the 'unknown'-exclusion bias; both are within the scope of a revision. If the authors cannot provide atomic-level checker validation or exclusion-rate analysis, the central claim of validity would remain unsubstantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the artifact: 8,573 biomedical questions from PMC across 29 categories, plus an automated FactScore-style pipeline with public code and data. That is a real contribution for anyone working on medical LLM evaluation. The paper is honest about its lineage: the atomic-unit decomposition is FactScore applied to a new domain, and the dataset construction and filtering are described in enough detail to reproduce. Credit where due: the dataset is sizable, the pipeline is fully automated, and the authors release both code and data. That alone makes this worth a look.\n\nThe soft spots are real but not fatal. The main validation is a Pearson correlation of 0.55 on 99 responses against human scores. That is moderate, and the paper overreads it as \"moderate to strong.\" No inter-annotator agreement is reported, so we do not know how reliable the human side is either. The paper also does not validate the pplx-API checker against any gold standard at the atomic-unit level; the correlation with human scores is the only evidence, and it is at the response level.\n\nThe more specific concern is the handling of \"unknown\" labels. The paper states in Section 5.3 that if any atomic unit is labeled unknown, the whole response is dropped from the DAHL Score calculation. That is a selection rule that can bias model comparisons if unknown labels occur unevenly across models, and the paper never reports how often this happens. This is a genuine soft spot, but it is fixable: report exclusion rates per model, or treat unknown as a third category instead of dropping the response.\n\nThe scaling claim (plateau past 7-8B) is based on visual inspection of Table 2 with no significance tests. That is a minor overreach; the pattern is plausible given Qwen-2 and Llama-3.1, but it is not established.\n\nCircularity is not a real issue here: the checker is separate from the evaluated models, and the labels come from a web-augmented API. Using gpt-4o to split gpt-4o's own output is a minor bias, not a load-bearing one.\n\nI would send this to peer review. The dataset and pipeline are useful, the limitations are addressable, and the paper's central idea holds up as a reasonable domain adaptation. The authors need to tighten the validation and report exclusion rates, but this is a solid starting point for the community. A serious referee can push on those points productively.","headline":"A genuinely useful biomedical hallucination benchmark dataset and pipeline, but the validation is thin and the unknown-label exclusion could bias every model ranking.","tokens_in":15367,"tokens_out":620,"would_cite":true,"duration_ms":7707,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that DAHL, an automated pipeline that splits long biomedical answers into atomic claims and checks each against search-grounded sources, produces a DAHL Score that tracks human hallucination judgments (Pearson r = 0.5508…","keywords":["hallucination evaluation","biomedical benchmark","long-form text generation","atomic factuality","DAHL Score","large language models","factual precision"],"falsifier":"A direct falsifier: on a random sample of responses, ask the Checker to label every atomic unit with a forced true or false choice instead of allowing 'unknown', then recompute the DAHL Score and its correlation with human judges; if the correlation drops substantially or model rankings change, the reported validity is an artifact of excluding ambiguous units.","tokens_in":14419,"feed_emoji":"🧬","tokens_out":7092,"duration_ms":68930,"temperature":0.7,"pith_summary":"This paper tries to make hallucination in long-form biomedical answers measurable without expensive human annotation. It introduces a benchmark of 8,573 questions built from biomedical research papers and an automated pipeline that splits each model response into atomic factual claims, checks each claim against online-sourced information, and averages the truth rate into a DAHL Score. The paper argues that this score tracks human judgments: on 99 responses the automated and human factuality scores correlate at 0.5508, with a p-value of 3.49 × $10^{{-9}}$. If valid, the score gives a cheap, repeatable way to rank models by factual reliability and to generate preference labels for alignment tuning in a high-stakes domain.","feed_headline":"Automated score matches humans on biomedical hallucination","feed_subtitle":"DAHL splits medical answers into atomic claims, checks each against web sources, and matches human ratings, r=0.55.","key_machinery":"The central object is the atomic unit: a sentence-sized piece of a model response that asserts one fact whose truth can be judged true or false. The pipeline works in three stages: preprocessing removes the prompt, duplicate sentences, and truncated text; a splitter model (the paper uses gpt-4o) breaks the response into atomic units; and a search-grounded API checker labels each unit true, false, or unknown. The DAHL Score for a response is the ratio of true atomic units to total atomic units, and the overall DAHL Score is the average of these ratios across responses. The atomic-unit representation is what lets the method credit a response for the claims it gets right even when other claims in the same answer are wrong.","core_discovery":"On the paper's own terms, the central discovery is that fact-conflicting hallucination in long-form biomedical text can be decomposed into atomic units and automatically scored by a splitter-and-checker pipeline, and that the resulting DAHL Score is statistically correlated with human factuality ratings. The authors report a Pearson correlation of 0.5508 between automated and human scores across 99 responses, and they further observe that larger models tend to hallucinate less, with diminishing returns beyond roughly 7 to 8 billion parameters. They position the DAHL Score as a potential efficient alternative to human-annotated preference labels, and they claim the automated dataset construction pipeline is scalable to other specialized domains.","pith_inferences":["The scaling-plateau claim is a statement about these eight models and this benchmark, not a general law; controlling for training-data overlap and question difficulty would be needed before generalizing it.","The score's usefulness as a preference label depends on checker agreement with human judgments on individual atomic units, not just on the response-level correlation, so unit-level agreement statistics would be a natural next measurement.","Because the checker consults live online sources, DAHL scores are not fully reproducible over time; fixing a dated document snapshot would make the benchmark a stable evaluation target.","If the checker's 'unknown' exclusions remove systematically harder or more ambiguous responses, the reported scores may be optimistic; recomputing scores with forced true/false labels would reveal the size of that bias."],"forward_implications":["The DAHL Score can be computed automatically on long-form biomedical answers, replacing costly human annotation for ranking models by factual reliability.","The same pipeline can produce continuous preference labels for alignment tuning, since each response receives a factuality score rather than a binary judgment.","Factual accuracy in biomedicine improves with model size up to roughly 7 to 8 billion parameters but shows diminishing returns beyond that, at least for the tested open models.","A temperature range of roughly 0.2 to 0.3 gives moderate hallucination performance across models, with no single universal optimal temperature.","The automated dataset construction from a public biomedical full-text archive can be extended to other specialized domains and updated as new papers appear."],"supporting_citations":[{"why":"Supplies the atomic-unit factual precision decomposition that DAHL adapts to long-form biomedical responses.","marker":"Min et al. 2023"},{"why":"Provides the 28 medical categories and the domain-specific hallucination baseline whose multiple-choice limitation DAHL addresses.","marker":"Pal et al. 2023"},{"why":"Supplies the Llama-3 models used both as evaluated systems and as the base LLM of the Checker.","marker":"Dubey et al. 2024"},{"why":"Provides the Gemma-2 models used in the size-scaling and temperature comparisons.","marker":"Team et al. 2024"},{"why":"Provides the Qwen-2 models whose 0.5B-to-72B size sweep supports the scaling-plateau claim.","marker":"Yang et al. 2024"},{"why":"Defines the input-, context-, and fact-conflicting hallucination taxonomy that fixes DAHL's scope to fact-conflicting hallucination.","marker":"Zhang et al. 2023"},{"why":"Offers the binary truthfulness-judgment baseline that DAHL positions against in favor of finer-grained scoring.","marker":"Lin et al. 2022"}],"fun_headline_variants":["Atomic claim checker scores biomedical hallucination like humans","DAHL splits claims to score LLM hallucinations, matches human ratings","Biomedical hallucination score matches humans by checking atomic claims","LLM factuality score: atomic claims, human-like ratings, scaling plateau","DAHL: automated hallucination score correlates with human judgments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the automated checker's true and false labels on atomic claims are accurate enough to stand in for human fact-checking, which the paper never directly validates against a gold standard.","fun_headline_variants_meta":{"raw":{"variants":["Atomic claim checker scores biomedical hallucination like humans","DAHL splits claims to score LLM hallucinations, matches human ratings","Biomedical hallucination score matches humans by checking atomic claims","LLM factuality score: atomic claims, human-like ratings, scaling plateau","DAHL: automated hallucination score correlates with human judgments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00049,"raw_usage":{"total_tokens":2365,"prompt_tokens":856,"completion_tokens":1509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1423}},"tokens_in":472,"tokens_out":1509,"duration_ms":12050,"temperature":1.0,"reasoning_tokens":1423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:50:07.492441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier: on a random sample of responses, ask the Checker to label every atomic unit with a forced true or false choice instead of allowing 'unknown', then recompute the DAHL Score and its correlation with human judges; if the correlation drops substantially or model rankings change, the reported validity is an artifact of excluding ambiguous units.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the atomic-unit factual precision decomposition that DAHL adapts to long-form biomedical responses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the input-, context-, and fact-conflicting hallucination taxonomy that fixes DAHL's scope to fact-conflicting hallucination."}],"review_version":1}