{"id":"12573149-4a3f-4237-8e3b-ff01f08c0b82","arxiv_id":"2608.03675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VetScore adds a harm-potential weighting to claim-level fact verification and matches veterinary expert judgments on a new 1,200-segment annotated dataset.","lead":"VetScore is a pipeline that checks each claim in a veterinary AI answer against the cited source text and gives more weight to claims that could harm a patient if wrong. It matches veterinary expert judgments well on a new annotated dataset, even with smaller AI models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Risk-weight exponent p=2.426 is hand-set and shared by the model and human combined gold; without sensitivity analysis the combined-score correlations may reflect a shared arbitrary weighting.","rationale":"The paper's main evidence is the component-level correlations with human experts across nine judge models, plus the combined-score columns. My read confirms the strongest_claim: the independent components are credible because they are evaluated on a human-annotated dataset, with multiple runs, a manual decomposition check, and ablations. The weakest spot is exactly the one the reader identified: the human gold 'combined' score is built with the same hand-set exponent p=2.426 that defines the model score. The footnote rationale (that harm level 5 should carry 50% of total weight under uniformly distributed levels) is a formal heuristic, not a behavioral validation of how veterinarians trade off harm against verification. Since Spearman correlation is not invariant to reweighting across claims within a segment, the high combined correlations could partly reflect agreement on the weighting formula rather than on clinically meaningful risk-adjusted verification. This is not an internal inconsistency, and the paper's promise to release code and data makes the concern testable. It is precisely the kind of assumption that should keep the verdict CONDITIONAL rather than ACCEPT. I therefore agree with the reader's weakest assumption and would keep the conditional verdict; no further downgrade is warranted on the current evidence.","tokens_in":30484,"tokens_out":5367,"duration_ms":61837,"concrete_test":"Recompute all combined-score columns of Table 3 for p ∈ {0, 1, 1.5, 2, 2.426, 3, 4, 5}, using the released annotations and the same pipeline as in §5 (same claims, same human aggregation, same judge models, n=5 runs). If the Spearman correlations and system-level rankings remain high and stable across this range (e.g., |Δρ| < 0.05 for Gemini 3.6 Flash and Gemma 4 31B), the hand-set p is not load-bearing. If correlations peak sharply near 2.426, vary by more than ~0.1 across plausible p values, or if the p that maximizes correlation differs across judge models, the combined-score claim is substantially an artifact of the shared weighting, and the component-level correlations should be treated as the primary evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in §3.2, Eqns. (1)-(4) and footnote 2: the combined score exponentiates harm weights with a hand-set p=2.426, and the same p is used to construct the human 'combined' gold score against which every combined column of Table 3 is correlated. This inserts the same arbitrary nonlinearity on both sides of the comparison. Spearman correlation is rank-based, so if p does not reflect how veterinarians actually trade off harm, the rankings produced by S_abs and S may be jointly distorted; the reported combined correlations (e.g., 0.740 for Gemini 3.6 Flash, 0.677 for Gemma 4 31B) could reflect agreement on the weighting convention rather than on clinically meaningful risk-adjusted verification. The paper does not vary p, does not compare against p=1 or equal weighting, and does not elicit a harm-tradeoff weighting from experts. The harm-potential gold itself has only moderate inter-annotator agreement (α=0.462), so the combined gold is noisy. The fact-verification and harm-potential component correlations are independent of p and are the most robust part of the paper; the concern is specific to the risk-weighted combined-score headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VetScore, a decompose-then-verify evaluation method for veterinary long-form QA with citations. The pipeline segments a generated answer, decomposes each segment into atomic claims, verifies each claim against the cited excerpts, scores each claim for its potential clinical harm on a 1–5 scale, and aggregates the two signals into a risk-adjusted score using an exponent p=2.426 (Eqs. 1–4). The authors construct a meta-evaluation dataset from 67 queries, six generator LLMs, 1,200 cited segments, and 4,986 claims, annotated by 18 veterinary experts. They evaluate nine judge models and report Spearman correlations up to 0.783 for fact verification, 0.763 for harm potential, and 0.740 for the absolute combined score, with the open-weight Gemma 4 31B reaching 0.724/0.721/0.677. Ablations, IRT-based annotator adjustment, consistency analysis, and token-efficiency analysis are also included.","tokens_in":30714,"tokens_out":5567,"duration_ms":59936,"significance":"If the central claim holds, VetScore offers a clinically interpretable, explainable, and risk-aware faithfulness signal for a high-stakes domain, and the expert-annotated dataset is a useful resource for future work. The component-level results—fact verification and harm potential scoring—are the most robust part of the paper: they are evaluated against independent human annotations and show strong correlations across multiple judge models, including open-weight ones. The paper also deserves credit for its explicit ablation study, use of IRT to handle annotator severity, and systematic analysis of decomposition errors. However, the paper's headline risk-weighted combined score is currently under-supported because the human 'combined' gold is constructed using the same hand-set exponent p that defines the model score, and no sensitivity analysis or holistic expert validation is provided. This limits the strength of the conclusions that can be drawn from the combined-score columns.","major_comments":[{"comment":"The combined-score validation is confounded by the shared hand-set exponent p. The paper fixes p=2.426 in footnote 2 and uses the same exponent to build both the model score and the human 'combined' gold against which the model is correlated. Since Spearman correlation is rank-based, any monotone transformation of the harm weights affects the model and human rankings in the same way. The reported combined correlations (e.g., 0.740 for Gemini 3.6 Flash, 0.677 for Gemma 4 31B) may therefore reflect agreement on the weighting convention rather than agreement on a clinically meaningful risk-adjusted verification score. The paper does not vary p, does not compare against p=1 or equal weighting, and does not elicit a harm-tradeoff weighting from experts. This is load-bearing because the risk-adjusted combined score is the paper's core contribution. Please add a sensitivity analysis over p and","section":"Section 3.2, Eqs. (1)–(4), Table 3"},{"comment":"The manuscript explicitly states in the Limitations that holistic segment-level scores were not annotated. The human combined gold is constructed by applying the same aggregation equations (Eqs. 2–4) to component human scores. This makes the combined-score correlation partly circular: the model and the gold share the same nonlinear weighting assumption. The limitation is acknowledged, but it is not reflected in the abstract or conclusion, where the combined score is presented as a headline result. A subset of segments with holistic expert risk-adjusted ratings would break the circularity; at minimum, the claims in the abstract and Section 7 should be tempered to component-level validation unless such evidence is added.","section":"Limitations, Section 5 (Meta-evaluation)"},{"comment":"The claimed advantage of risk weighting is not visible in the normalized local and global combined scores. For Gemini 3.6 Flash, verification-only achieves rho=0.770 (local) and 0.788 (global), compared with VetScore's 0.760 and 0.772; the differences are small and sometimes in the opposite direction from the headline. The only column where risk weighting improves the correlation substantially is the absolute combined score (0.666 vs. 0.740 for Gemini 3.6 Flash). However, S_abs is unbounded and sensitive to segment length and claim count, so it is not a clean measure of risk-adjusted verification. The paper should either explain why the normalized scores are the appropriate metric despite showing no benefit from risk weighting, or restrict the risk-weighting claim to the absolute score with appropriate caveats.","section":"Table 3, Verification-only rows"},{"comment":"The harm-potential annotation has only moderate inter-annotator agreement (Krippendorff's alpha=0.462). While the IRT-based adjustment in Appendix G is a reasonable and well-described remedy, the noise in the harm gold propagates into the combined gold and therefore into the combined-score correlations. This is acknowledged only implicitly. The combined correlations should be interpreted with this noise in mind, and the paper would be stronger if the sensitivity of the combined score to the harm-gold quality were analyzed, e.g., by recomputing the combined correlation under alternative aggregations of the raw harm annotations.","section":"Section 4.3, Task 2"}],"minor_comments":[{"comment":"The notation in Eq. (1) reuses the index i for both the segment and the claims within the segment. Use a separate index (e.g., j) for claims to avoid confusion, consistent with the problem statement.","section":"Eq. (1)"},{"comment":"The text in Appendix J.1 swaps the references to Figure 15 and Figure 16. It states that Figure 16 shows human scores and Figure 15 shows LLM judge scores, but the figures are captioned the other way around.","section":"Appendix J.1"},{"comment":"The abstract says 'Code and data will be released' while the contributions list says 'We make our meta-evaluation dataset publicly available.' Please make the release status and timeline consistent, and note whether the annotation interfaces and model outputs will also be released.","section":"Abstract / Section 4"},{"comment":"Minor grammar issue: 'how well are generated claims supported' should be 'how well generated claims are supported.'","section":"Abstract"},{"comment":"The table caption introduces 'Combined (local)' and 'Combined (global)' but the main text does not define which equations produce these variants. Please add a reference to Eqs. (2) and (3) in the caption or surrounding text.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The strongest citable results are the component-level correlations and the expert-annotated dataset. The combined risk-weighted score needs either a sensitivity analysis over p or a holistic expert gold before the headline claim is fully supported. If the authors cannot provide such evidence, I would recommend they reframe the contribution around component scores and risk-aware explainability rather than the specific p=2.426 aggregate. The paper is within the journal's scope and the methodology is generally sound, so I see this as a fixable revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. It is the first decompose-and-verify fact verification pipeline I know of that weights claims by clinical harm potential before aggregation, and it ships a new expert-annotated veterinary meta-evaluation dataset (1,200 segments, ~5,000 claims, 18 experts). That is a real contribution. The component-level evaluation is the strong part: fact verification and harm scoring each correlate with expert judgments at rho ~0.72–0.78 with good judge models, and the ablations are thoughtful.\n\nThe soft spot is the combined score. The aggregation uses a hand-set exponent p=2.426, chosen so a harm score of 5 carries 50% of the weight under uniform harm levels. The same p is used to build the human combined gold against which the final score is correlated. That puts the same arbitrary nonlinearity on both sides of the Spearman comparison, so the headline combined correlations (0.74 for Gemini 3.6 Flash, 0.68 for Gemma 4 31B) may partly reflect agreement on the weighting convention rather than clinically meaningful risk adjustment. The paper does not vary p, does not compare to p=1 or equal weighting, and does not elicit a harm-tradeoff from experts. The limitation section concedes there is no holistic expert score; that is exactly the missing check. So I treat the combined-score numbers as unproven, not wrong.\n\nSecond soft spot: code and data are promised but not released. The IAA on harm scoring is moderate (alpha 0.462), which is not terrible for the construct but does add noise to the gold. And the \"small judge models\" claim is uneven: Gemma 4 31B is genuinely competitive, but GLM-4.7 Flash and Nemotron 3 Nano are far below (rho 0.4–0.45 on verification). The paper does not hide this, but the abstract's phrasing overstates it.\n\nNone of this is fatal. The component correlations are independent of p, the dataset is new, the ablations are real, and the writing is clear and honest about limitations. The paper is a solid application/extension of the established decompose-then-verify paradigm, not a new paradigm.\n\nWho is this for: anyone building citation-faithfulness metrics for medical QA, or evaluating LLM outputs where claim severity matters. It deserves a serious referee. My recommendation: send it to peer review and ask for (a) sensitivity analysis over p, (b) comparison against a holistic expert score or at least a simpler weighting, and (c) actual data/code release before publication.","headline":"Solid risk-weighted verification pipeline for veterinary QA with a genuinely useful dataset, but the combined-score headline leans on a hand-set weight shared with the human gold.","tokens_in":31218,"tokens_out":2674,"would_cite":true,"duration_ms":27102,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VetScore claims an automated, risk-weighted faithfulness score can match veterinary expert judgments on cited long-form answers, reaching a 0.783 correlation with small judge models.","keywords":["fact verification","risk-weighted evaluation","harm potential","veterinary medicine","long-form question answering","citation faithfulness","LLM-as-judge","decompose-then-verify"],"falsifier":"Re-run the meta-evaluation with p=1 (plain harm-weighted average) and with p fitted to maximise expert agreement; if the rank correlation at p=1 does not fall, or if a fitted p differs substantially from 2.426, then the specific risk-weighting mechanism is not what drives the reported agreement.","tokens_in":30353,"feed_emoji":"🐾","tokens_out":6392,"duration_ms":66801,"temperature":0.7,"pith_summary":"VetScore is an evaluation method for veterinary long-form question-answering systems that produce answers with citations. Its central claim is that the faithfulness of an answer to its cited excerpts can be scored automatically in a way that weights each claim by how much harm a wrong claim could cause, and that this score tracks the judgments of veterinary experts. On a new expert-annotated meta-evaluation dataset of 1,200 segments and 4,986 claims, the method reaches rank correlations up to 0.783 for fact verification, 0.763 for harm potential, and 0.740 for the combined risk-adjusted score, with the open-weight Gemma 4 31B model close behind at 0.724, 0.721, and 0.677. The claim matters because high-stakes medical answers carry unequal risk—a wrong drug dose is more dangerous than a wrong study year—and existing fact-verification metrics treat every claim equally.","feed_headline":"VetScore weighs harm to catch unverified vet advice","feed_subtitle":"Decompose-and-verify pipeline matches veterinary experts (0.78 correlation) and flags risky claims.","key_machinery":"The load-bearing object is the risk-adjusted combined score S(Y')=1−p_global(Y'), where p_global is the harm-weighted penalty: each claim contributes w_i^p(1−v_i) to the numerator and Σ w_j^p to the denominator, with v_i the binary verification label, w_i the 1–5 harm potential, and p=2.426 chosen so a harm score of 5 accounts for about half the weight when harm levels are uniformly distributed. The decompose-then-verify architecture with separate harm scoring is the mechanism that makes the weighting possible; the paper contrasts it with verification-only and end-to-end baselines.","core_discovery":"The paper introduces a four-step pipeline: split an answer into citation segments; decompose each segment into atomic claims using an LLM with conversational context; score each claim twice—once for whether the cited excerpts support it (binary) and once for its harm potential on a 1–5 scale; then aggregate the two signals into a risk-adjusted verification score. The aggregation raises each harm score to a fixed exponent p=2.426 so that high-harm claims dominate the penalty, and reports the fraction of total weighted harm potential that remains unverified. The authors argue that this risk-weighting is what makes the score clinically informative, and they validate it by correlating both the c","pith_inferences":["If the risk exponent were re-estimated on held-out expert data instead of being hand-set, the pipeline could adapt across specialties; testing a flat p would reveal whether the exact value matters.","Because the score only measures claims the system actually states, a model that omits a key warning could score well; explicit omission-aware penalties would close that gap.","The same harm-weighted decomposition could transfer to human medical QA, but the 1–5 rubric would need re-anchoring to human clinical decisions before use.","Comparing VetScore against a simple harm-weighted average (p=1) would isolate whether the concave weighting, rather than the decomposition, drives expert agreement."],"forward_implications":["An automated VetScore run can flag individual unverified high-harm claims (e.g. a wrong dose) in real time, giving practitioners a threshold-based safety filter rather than a single opaque number.","Deployment with open-weight models like Gemma 4 31B is viable, with only about 0.06 lower correlation than the best proprietary judge on the combined score.","Verification-only scoring without harm weighting loses alignment with expert absolute combined scores, so the risk-weighting component is doing measurable work.","End-to-end single-prompt scoring, without decomposition, correlates less than half as well with experts, showing the multi-step structure is necessary.","The component scores (verification, harm) each carry explainability as natural-language justifications, so the pipeline doubles as an audit trail."],"supporting_citations":[{"why":"Supplies the decompose-then-verify paradigm that VetScore extends with harm weighting.","marker":"Min et al., 2023"},{"why":"SAFE baseline for long-form factuality that also decomposes then verifies claims.","marker":"Wei et al., 2024"},{"why":"VeriScore claim-filtering approach contrasted; filters to verifiable claims, which the authors reject for medical use.","marker":"Song et al., 2024"},{"why":"MedScore; documents that general factuality metrics produce invalid claims in the medical domain.","marker":"Huang et al., 2025"},{"why":"Provides decontextualization desiderata that shape the claim-decomposition prompt.","marker":"Gunjal and Durrett, 2024"},{"why":"Classifies claims by relevance to the query; the authors contrast their harm-based weighting with query-relative importance.","marker":"Wanner et al., 2025"},{"why":"Partial credit model used to convert raw harm annotations into rater-adjusted gold scores.","marker":"Masters, 1982"},{"why":"Item response theory approach used to build the evaluation scale for harm potential.","marker":"Lalor et al., 2016"}],"fun_headline_variants":["Risk-weighted scoring flags unverified vet claims","VetScore: harm-aware verification for vet answers","New score weighs harm to catch unsupported vet advice","Decompose, verify, weigh harm: vet QA scoring","0.78 expert match: risk-weighted vet claim check"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the hand-set exponent p=2.426, which makes a harm score of 5 carry about half of the total weight, captures how veterinarians actually trade off harm; the same exponent is used to build the human gold score that VetScore is measured against, so if that weighting is wrong, the combined-score validation weakens.","fun_headline_variants_meta":{"raw":{"variants":["Risk-weighted scoring flags unverified vet claims","VetScore: harm-aware verification for vet answers","New score weighs harm to catch unsupported vet advice","Decompose, verify, weigh harm: vet QA scoring","0.78 expert match: risk-weighted vet claim check"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1183,"prompt_tokens":675,"completion_tokens":508,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":419,"tokens_out":508,"duration_ms":5667,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:58:10.925181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the meta-evaluation with p=1 (plain harm-weighted average) and with p fitted to maximise expert agreement; if the rank correlation at p=1 does not fall, or if a fitted p differs substantially from 2.426, then the specific risk-weighting mechanism is not what drives the reported agreement.","supporting_citations":[],"review_version":1}