{"id":"3ec58da2-09ac-4323-9b56-344781b87d82","arxiv_id":"2501.03545","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ICAT automatically scores long-form text on both factual accuracy and diverse aspect coverage, with its best variant correlating with human judgments better than standard n-gram and embedding metrics.","lead":"This paper introduces ICAT, a framework that checks both how factual a long-form LLM answer is and how well it covers the different aspects a good answer should include. It shows that the method's coverage scores correlate with human judgments on 50 web-search queries, while traditional metrics like BLEU and ROUGE mostly fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human gold standard and ICAT share the same fixed TREC subtopic checklist, so the reported correlation may measure checklist agreement rather than coverage of the true aspect space.","rationale":"The reader's weakest assumption is the completeness and validity of TREC subtopics as ground-truth aspects; I agree and sharpen it. The central claim requires that the aspect set used for coverage is a valid representation of the space of aspects a long-form answer should cover. The TREC subtopics were designed for search result diversification, not long-form answer completeness. More importantly, the human evaluation is not independent of this list: annotators are given the list and asked to check presence. This makes the reported correlation a checklist-agreement measure, not a validation of coverage over the true aspect space. The paper's own admission that auto-generated topics are more comprehensive than TREC subtopics supports the concern. Other issues, such as missing confidence intervals, same-family LLM bias, and the factuality-filter mismatch between ICAT and human annotations, are real but secondary: confidence intervals would quantify uncertainty without changing the construct, same-family bias is acknowledged and mainly affects ICAT-A, and the factuality mismatch could be tested separately. The proposed expanded-aspect check would settle whether the shared checklist drives the reported correlation. Therefore the reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":15850,"tokens_out":5601,"duration_ms":52937,"concrete_test":"Build an expanded gold aspect set for the same 50 queries: have fresh annotators freely list the aspects a complete long-form answer should cover, then merge these with the TREC subtopics (or use human-validated LLM-generated aspects, as the paper's Table 2 note suggests). Recompute human coverage scores and ICAT-S/ICAT-A coverage scores against this expanded set and re-estimate the correlations in Table 2. If ICAT-S-human correlation drops substantially or becomes nonsignificant, the headline result is an artifact of the shared TREC checklist; if it remains strong, the concern is bounded. As a secondary check, ask annotators to flag missing aspects for each response; a high miss rate would confirm the checklist incompleteness directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central validation in Section 6 computes both the human coverage scores and ICAT coverage scores against the same ground-truth TREC Web Track subtopic list (Section 5). Annotators were asked to mark whether each listed aspect is present in the response and to highlight evidence; they were not asked to judge overall completeness or to propose missing aspects. Thus the human 'gold standard' is not an independent measure of diverse factual coverage: it is a binary check on a fixed, search-diversification-oriented checklist. ICAT-S uses the same checklist as its aspect set, so the reported Pearson rho=0.422 / Spearman 0.446 (Table 2) can be read as inter-annotator agreement between a human and an LLM on a shared checklist, not as evidence that ICAT recovers the aspects a good long-form answer should cover. The paper itself notes under Table 2 that automatically generated aspects were 'more comprehensive and higher quality' than TREC subtopics, acknowledging the reference list is incomplete. If the true aspect space is larger or different, both the human scores and ICAT scores share the same incomplete denominator, and the correlation is not informative about coverage of unlisted aspects. This is the load-bearing gap for the claim that ICAT measures coverage of diverse factual information in long-form generation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ICAT, a reference-free evaluation framework for long-form text generation. ICAT decomposes a generated response into atomic claims, verifies each claim by retrieving supporting evidence from a corpus (ClueWeb09) or the web and applying an NLI model, and then computes a coverage score as the fraction of query aspects supported by factually grounded claims. Three variants are instantiated: ICAT-M (manual aspects and manual document-aspect alignment), ICAT-S (manual aspects with LLM-based claim-aspect alignment), and ICAT-A (LLM-generated aspects with LLM-based alignment). The framework is evaluated on 50 TREC Web Track queries, comparing ICAT coverage scores against AMT annotators' judgments on the TREC subtopic lists, and on four LLM generators (GPT-4, Llama-3-70B, Mixtral-8x22B, Openchat 3.5). The best reported correlation is ICAT-S with Llama-3.1-70B (Pearson 0.422, Spearman 0.446 in the corpus setting); BLEU, ROUGE, and METEOR show near-zero correlation, while BERTScore-recall reaches 0.291. The paper also presents a beta-weighted harmonic combination of factuality and coverage and component-level analyses of claim generation, topic generation, and coverage alignment.","tokens_in":16049,"tokens_out":7035,"duration_ms":61042,"significance":"If the central validation were independent of the fixed TREC subtopic checklist, ICAT would be a practically useful, modular reference-free evaluator for long-form generation. The paper's strengths include a clean decomposition into atomic claims with retrieval-based grounding, three clearly specified variants that separate the effect of manual versus automatic aspects and alignment (Table 1), a released open-source implementation, component-level human evaluations (Tables 6-8), and a fair comparison showing that lexical overlap metrics carry near-zero signal for coverage. The reported coverage-prediction precision and recall against human judgments for ICAT-S (Table 8: 0.903 and 0.798) are encouraging. However, the contribution's central claim hinges on a validation whose human gold standard shares the same fixed aspect list as the metric, and the fully automatic variant's correlations are weak; these issues need to be addressed before the claim that ICAT measures coverage of diverse factual information can be accepted.","major_comments":[{"comment":"The validation protocol computes both the human coverage judgments and the ICAT coverage scores against the same fixed TREC Web Track subtopic list. The AMT task (Section 6, Figure 4) asked annotators to mark whether each listed aspect is present and to highlight evidence; it did not ask them to judge overall completeness or propose missing aspects. ICAT-S aligns claims to exactly the same list. The reported Pearson rho = 0.422 and Spearman rho = 0.446 can therefore be interpreted as human-LLM agreement on a shared checklist rather than as evidence that ICAT recovers the aspects a good long-form answer should cover. The TREC subtopics were designed for web-search result diversification, not for long-form generation, and the paper's own note under Table 2 states that auto-generated topics were 'more comprehensive and higher quality' than the TREC subtopics. Because both sides of the correlation use the same incomplete denominator, the result is not informative about coverage of unlisted aspects.","section":"Section 5 and Section 6, Table 2"},{"comment":"The fully automatic variant ICAT-A, which is the only one that does not require manually supplied aspects, reaches only Pearson rho = 0.246 in the corpus setting and rho = 0.055 in the web setting with Llama-3.1-70B as the coverage model; the web correlation is effectively zero and no significance test is reported for it. The Section 7 statement that automatic evaluation approaches 'maintain comparable performance' is therefore supported only by ICAT-S, which assumes ground-truth aspects are available. Since the abstract and introduction motivate ICAT as an automated evaluator of coverage, the authors should either qualify the claim or show that ICAT-A's low correlation is an artifact of the incomplete TREC aspect set rather than a property of the metric.","section":"Table 2"},{"comment":"All correlations are computed on 50 queries, yet no confidence intervals are reported. With n = 50, the best Pearson rho = 0.422 has a wide confidence interval, and the difference between ICAT-S (0.422) and ICAT-A (0.246) may not be statistically significant. Report bootstrap confidence intervals for every row of Table 2 and a significance test for the difference between ICAT-S and the best traditional metric (BERTScore-recall, 0.291).","section":"Section 6"},{"comment":"The aspect-generation and aspect-claim alignment components use Llama-3.1 models, while the set of evaluated generators includes Llama-3-70B; the paper itself acknowledges that using the same or similar LLM for aspect generation and response evaluation can inflate performance. The near-zero web correlation of ICAT-A (0.055) is consistent with this bias rather than with robust coverage measurement. The authors should test at least one configuration where the aspect and alignment model is from a different model family than the evaluated generator, or where the aspects are curated independently of both the generator and the TREC list.","section":"Section 4.2 and Limitations"}],"minor_comments":[{"comment":"The phrase 'corelation' in the third paragraph should be corrected to 'correlation'.","section":"Introduction"},{"comment":"The phrase 'rank-bsaed' in the first paragraph should be corrected to 'rank-based'.","section":"Section 6"},{"comment":"The header notation 'ICAT-M1', 'ICAT-S1', and 'ICAT-A1' is not defined in the caption or the text; clarify that the subscript denotes the default beta = 1 value.","section":"Table 3"},{"comment":"The symbols T_O and T_Q are introduced with subscript-like notation but are rendered ambiguously in the coverage-score equation; use a consistent mathematical notation for the claim-to-aspect and query-to-aspect functions.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: ICAT is a well-built, reusable coverage metric, and its best variant correlates moderately with human aspect-coverage judgments. But the validation is anchored to the same TREC subtopic checklist that both the human annotators and the metric score against, so the headline number is checklist agreement rather than proof that ICAT captures the true space of aspects.\n\nWhat's new is the composition: atomic claims from FActScore, retrieval-based grounding from FActScore/VERISCORE, plus an explicit claim-aspect alignment that yields a coverage score, with three variants spanning fully manual to fully automatic. That's a pragmatic design and the paper executes it carefully. The component-level analysis is better than most metric papers bother with: human-evaluated claim generation (fact precision 0.838), topic generation precision of 0.966, coverage precision/recall of 0.903/0.798, and NLI benchmark numbers. The code is public, and the human study is careful (three annotators, Fleiss's kappa = 0.829, majority voting). The comparison against traditional metrics is useful: BERTScore-recall reaches 0.291, well below ICAT-S's 0.422 (corpus) and 0.525 (web).\n\nThe stress-test concern is real, though I'd frame it as an external-validity limit rather than a fatal circularity. The annotators ticked which listed TREC subtopics appeared in the response; they weren't asked whether the list was complete. Both human and ICAT-S coverage scores share the same denominator, so the correlation is inter-method agreement on a fixed checklist. That's a legitimate validation target—automating human checklist judgments is useful—but the title's 'diverse factual information' claim is only as strong as the checklist. The authors effectively concede this under Table 2, where they note that auto-generated aspects looked 'more comprehensive and higher quality' than the TREC subtopics. I'd want a missing-aspect analysis or a human completeness study before buying the broader claim.\n\nOther soft spots are proportionately minor: n=50 with no confidence intervals; the fully automatic ICAT-A variant is the weakest at 0.246 (corpus) and 0.055 (web); the same-family LLM bias in ICAT-A is acknowledged in the Limitations section but unquantified; and the lexical baselines were given concatenated aspect text as a reference, so their near-zero correlations are not a headline result.\n\nWho this is for: anyone building LLM evaluation metrics or thinking about coverage as a training reward. It deserves a serious referee. I'd send it to review and ask for confidence intervals, the missing-aspect analysis, and a human study on the generated aspect set.","headline":"A well-engineered coverage metric whose best variant correlates moderately with human judgments on a shared TREC checklist; the framework is solid and reusable, but the broader 'diverse factual coverage' claim rests on that checklist's completeness.","tokens_in":16631,"tokens_out":6620,"would_cite":true,"duration_ms":57925,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-form answers should be judged on whether their verified facts cover all expected aspects, and the paper introduces ICAT, a retrieval-based framework that measures exactly that by decomposing outputs into atomic claims, grounding each…","keywords":["long-form text generation","factual coverage","evaluation metrics","atomic claims","aspect coverage","LLM evaluation","retrieval-based evaluation","text diversification"],"falsifier":"Re-run the human coverage study on the same queries but with an independently constructed aspect list, such as aspects free-listed by annotators or generated without the TREC subtopic set, and compare ICAT-S against human scores under both aspect sets. If the correlation with human judgments drops substantially when the aspect set changes, the reported alignment depends on the particular TREC subtopics rather than on coverage of diverse information itself.","tokens_in":15585,"feed_emoji":"✅","tokens_out":9715,"duration_ms":75062,"temperature":0.7,"pith_summary":"The paper argues that evaluating long-form text generation requires more than checking factual accuracy: a good answer must also cover the diverse aspects a user or topic implies. It introduces ICAT, which splits an LLM output into atomic claims, verifies each claim by retrieving supporting evidence from a knowledge source, and then scores how many of the expected aspects are covered by the verified claims. Across three implementations that differ in whether aspects and claim-aspect alignment come from humans or LLMs, the variant with human-provided aspects and LLM-based alignment (ICAT-S) correlates with human coverage judgments at Pearson's rho = 0.422 and Spearman's rho = 0.446 on ClueWeb/TREC queries. Traditional lexical metrics such as BLEU and ROUGE show negative or near-zero correlation with the same human judgments, which the paper reads as evidence that coverage of diverse information is a distinct quality that needs its own metric.","feed_headline":"Automated coverage metric matches human judges on long answers","feed_subtitle":"ICAT verifies each claim against a source, then checks expected aspects; BLEU and ROUGE fail at this.","key_machinery":"The load-bearing object is the ICAT score itself, a pair of ratios combined by a beta-weighted harmonic mean. Atomic claim generation turns each long output into decontextualized standalone facts; claim grounding uses dense retrieval followed by a natural language inference model to keep only claims supported by the knowledge source; aspect coverage maps those grounded claims to a query aspect list, either through aspect-annotated documents (ICAT-M), an LLM aligner (ICAT-S), or an LLM-generated aspect set (ICAT-A). The machinery's power is that it decomposes completeness into a checkable claim-level pipeline, making both factuality and coverage inspectable claim by claim.","core_discovery":"The central claim is that coverage of diverse factual information can be measured automatically by decomposing a response into atomic claims, grounding each claim through retrieval and natural language inference, and aligning the grounded claims with a list of expected aspects. The paper defines a factuality score $S_{\\mathrm{fact}} = |C_T|/|C|$ and a coverage score $S_{\\mathrm{coverage}} = |\\{T_O(c,K): c \\in C_T\\} \\cap T_Q(x)| / |T_Q(x)|$, combined as a weighted harmonic mean $ICAT_\\beta$. Its experiments show ICAT-S, using ground-truth TREC subtopics and Llama-3.1-70B for claim-aspect alignment, correlates strongly with human annotators (Pearson's rho = 0.422, p < 0.01; Spearman's rho = 0.446, p < 0.01). By contrast, BLEU, ROUGE, METEOR, and most BERTScore variants fail to track human coverage judgments. The paper also reports that web-based retrieval raises factuality scores across all tested LLMs but that coverage improvements are modest, and that the $\\beta$ parameter trades factuality against coverage in the combined score.","pith_inferences":["The paper's own observation that ICAT-A's automatically generated aspects look more comprehensive than the TREC subtopics suggests the reported correlations may underestimate how well the framework could perform with a better aspect target; testing ICAT-A against human judgment on richer aspect sets would settle this.","Because ICAT-A relies on the same or similar LLMs for aspect generation and response evaluation, responses sharing that model's biases could be scored as more complete; the paper flags this circularity but does not measure its size.","A natural extension is to apply the same claim-level coverage pipeline to domain-specific long-form tasks, such as clinical or policy writing, where the aspect list is defined by guidelines rather than search subtopics and the evidence corpus is a trusted authority."],"forward_implications":["Because ICAT-S tracks human coverage judgments at moderate-to-strong correlation, it can replace expensive human annotation when comparing LLMs on long-form answers.","BLEU, ROUGE, and METEOR are unsuitable for coverage evaluation: their correlations with human coverage judgments are negative or near zero, so they should not be used as proxies for completeness.","The beta parameter lets an evaluator weight factuality against coverage; for beta less than 1, GPT-4 and Mixtral lead, while for beta greater than 1, GPT-4 and Llama lead on coverage.","Switching from corpus-based to web-based retrieval increases factuality scores for every tested LLM, while coverage gains are smaller, indicating that coverage depends more on the generator than on the retrieval source.","The modular design supports adaptation to domains where aspect lists or evidence corpora exist, and ICAT's scores could serve as a reward signal for training models to produce more complete outputs."],"supporting_citations":[{"why":"supplies the atomic-claim decomposition and factuality-precision approach that ICAT's factuality score extends.","marker":"(Min et al., 2023)"},{"why":"provides the claim granularity and verifiability standards used when training and evaluating ICAT's claim generator.","marker":"(Song et al., 2024)"},{"why":"provides the TREC Web Track topics and subtopics that serve as the ground-truth aspect lists in ICAT-M and ICAT-S.","marker":"(Clarke et al., 2009, 2010, 2011, 2012)"},{"why":"supplies the ClueWeb09 collection used as the retrieval corpus for claim grounding.","marker":"(The Lemur Project, 2009)"},{"why":"supplies the spam filter used to clean the ClueWeb corpus before retrieval.","marker":"(Cormack et al., 2011)"},{"why":"supports the automatic LLM-based query aspect generation used in ICAT-A.","marker":"(Samarinas et al., 2022)"},{"why":"defines BLEU, the lexical metric whose near-zero correlation with human coverage judgments is the negative result ICAT is contrasted against.","marker":"(Papineni et al., 2002)"},{"why":"defines ROUGE, another lexical baseline that fails to track human coverage judgments.","marker":"(Lin, 2004)"}],"fun_headline_variants":["Forget BLEU: ICAT verifies each claim and powers coverage scoring","ICAT matches human judges on coverage of long-form facts","New metric: decompose into claims, verify, then score diversity","Automatic fact coverage scoring for long answers beats ROUGE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that the TREC Web Track subtopics, originally designed for web search result diversification, are a complete and valid set of aspects for a long-form answer; if the true space of aspects differs, both the human gold standard and the ICAT scores are anchored to an incomplete target.","fun_headline_variants_meta":{"raw":{"variants":["Forget BLEU: ICAT verifies each claim and powers coverage scoring","ICAT matches human judges on coverage of long-form facts","New metric: decompose into claims, verify, then score diversity","Automatic fact coverage scoring for long answers beats ROUGE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2398,"prompt_tokens":972,"completion_tokens":1426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1352}},"tokens_in":588,"tokens_out":1426,"duration_ms":11696,"temperature":1.0,"reasoning_tokens":1352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:40.142854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the human coverage study on the same queries but with an independently constructed aspect list, such as aspects free-listed by annotators or generated without the TREC subtopic set, and compare ICAT-S against human scores under both aspect sets. If the correlation with human judgments drops substantially when the aspect set changes, the reported alignment depends on the particular TREC subtopics rather than on coverage of diverse information itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the claim granularity and verifiability standards used when training and evaluating ICAT's claim generator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the ClueWeb09 collection used as the retrieval corpus for claim grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supports the automatic LLM-based query aspect generation used in ICAT-A."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines ROUGE, another lexical baseline that fails to track human coverage judgments."}],"review_version":1}