{"id":"7fe8eca2-4f30-49e7-b148-607154e668de","arxiv_id":"2411.16391","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes HCAT, a human-calibrated evaluation framework for RAG models, combining automated tests, embedding metrics, and conformal prediction.","lead":"This paper outlines a framework for testing and validating retrieval-augmented language models, mainly for banking applications. It combines automatic test generation, embedding-based scoring, and human calibration to make machine evaluation of model answers more transparent and reliable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HCAT's reliability claim depends on exchangeability between stratified synthetic test queries and real production queries; Section 5.2's conformal coverage guarantee is not shown to survive distribution shift, and no empirical evidence is provided.","rationale":"Read in good faith, the paper is an overview and proposal of a validation framework, not a report of a deployment. The central claim that HCAT is a 'practical and reliable solution' would require demonstrating that (a) the automatically generated synthetic queries are representative of real usage, (b) the embedding/NLI metrics after calibration track human judgments across topics and query types, and (c) the conformal prediction sets actually achieve their stated coverage in deployment. The paper provides none of these. The most load-bearing and technically precise gap is (c): Section 5.2's split conformal prediction guarantees coverage only under exchangeability, and HCAT's calibration samples come from stratified synthetic queries rather than the production query distribution. This is not a minor implementation detail; in banking, the stated confidence levels and prediction sets are what a validator would rely on for regulatory sign-off. The reader's verdict of CONDITIONAL is appropriate: the ideas are plausible and well structured, but the empirical core is missing. I do not see an internal inconsistency that would force rejection; the mathematical blemishes (σ(D) vs σ(z) in Sec 3.2.2; average pairwise distance labeled Wasserstein in Sec 3.3.2) are real but can be corrected without destroying the framework, and Stage 1 calibration could absorb monotone reparameterizations. A held-out-stratum and real-query coverage experiment would settle whether the exchangeability concern lands. If coverage fails, the central 'reliable solution' claim should be weakened to 'framework with unvalidated coverage guarantees'.","tokens_in":13871,"tokens_out":7107,"duration_ms":71180,"concrete_test":"Run a held-out-stratum experiment on a real banking RAG corpus: generate queries with the Section 2 pipeline, obtain human labels, fit Stage 1 logistic calibration and Stage 2 split conformal on one topic stratum (or on synthetic queries only), then evaluate coverage of the 90% prediction sets on (i) held-out topics from the same synthetic distribution and (ii) a small set of real production queries. If empirical coverage drops below the nominal level on either, the exchangeability assumption is violated and the reliability claim needs qualification or a distribution-shift-robust calibration method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that HCAT provides a 'practical and reliable solution' for deployment depends on the calibrated evaluation scores and conformal prediction sets produced in Section 5 being trustworthy for real queries. Section 5.2 correctly notes that split conformal prediction assumes data exchangeability, but the calibration and test data in HCAT are generated by stratified sampling of the document collection (Section 2), not sampled from the deployment query distribution. When production queries shift in topic, phrasing, or difficulty relative to the synthetic stratum distribution, the non-conformity score quantile q̂ computed on the calibration set no longer bounds the test non-conformity scores, so the advertised 1−α coverage is not guaranteed. The paper neither states this limitation nor offers a distribution-shift-robust procedure (e.g., weighted conformal or domain adaptation). Because the central claim is about reliable deployment in high-stakes banking, this gap is load-bearing: the framework's uncertainty quantification and regulatory assurances are exactly what would be relied upon, and the paper provides no empirical demonstration that coverage holds even within a single document collection across topics and query types. Secondary but related: Section 3.2.2 maps decision-boundary distance D to probability via σ(D) although the model's own logit is z = ‖w‖D, and Section 3.3.2 calls the average pairwise embedding distance the Wasserstein distance when it is only an upper bound under the independent coupling; both are correctness blemishes in the proposed metrics, but the exchangeability gap is the more fundamental threat to the reliability claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Human-Calibrated Automated Testing (HCAT), a framework for evaluating and validating retrieval-augmented generation (RAG) systems in high-stakes domains such as banking. The framework has five main components: automatic test generation through topic modeling and stratified sampling; embedding- and NLI-based functionality metrics (context relevancy, groundedness, completeness, answer relevancy); risk and safety metrics (toxicity, bias, privacy); a two-stage human calibration procedure combining probability calibration with split conformal prediction; and robustness testing with marginal and bivariate weakness analysis. The manuscript is explicitly an overview: it describes the components at a conceptual and formulaic level but reports no implementation, no dataset, no human annotation study, and no experimental evaluation.","tokens_in":14232,"tokens_out":5180,"duration_ms":52539,"significance":"The paper addresses a genuine and timely need: transparent, scalable evaluation of RAG systems in regulated industries. Its high-level architecture is sensible, and the emphasis on explainable metrics, human calibration, and uncertainty quantification is well placed. If the framework were implemented and validated, it could be a useful practical contribution. However, as submitted, the contribution is a proposal. The central claims of the abstract and conclusion, that HCAT provides a 'practical and reliable solution' whose evaluations 'align with human judgments,' are not demonstrated anywhere in the manuscript. The paper contains no machine-checked proofs, no reproducible code or data, and no falsifiable experimental results.","major_comments":[{"comment":"The paper's central claim is that HCAT produces evaluations that are aligned with human judgments and suitable for deployment decisions, but no empirical evidence is provided. Section 5 describes probability calibration and conformal prediction, yet there is no dataset, no human labels, no fitted calibration model, no comparison of machine scores with human judgments, and no evaluation of conformal coverage. The limitations listed in Section 7 do not include the absence of validation. For a framework paper to support the stated reliability claims, it needs at least one concrete case study on a RAG system with human-annotated outputs, calibration results, and an assessment of whether the calibrated metrics and prediction sets behave as claimed.","section":"Abstract, Sections 5 and 7"},{"comment":"The conformal prediction guarantee is stated under an exchangeability assumption, but the calibration and test data in HCAT are generated by stratified sampling of the document collection (Section 2), not sampled from the deployment query distribution. If production queries differ in topic, phrasing, or difficulty, the non-conformity score quantile computed on the synthetic calibration set does not in general bound the non-conformity scores on production queries, so the advertised 1-alpha coverage is not guaranteed. The paper acknowledges the exchangeability assumption but does not state this limitation or propose a distribution-shift-robust alternative (e.g., weighted conformal prediction). Since the deployment claim is the basis for the paper's relevance, this gap is load-bearing.","section":"Section 5.2 with Section 2"},{"comment":"The quantity defined after the optimal transport formula as the average of all pairwise distances, W(C,A) = (1/(nk)) sum d(ci,aj), is not the Wasserstein distance in general. It is the expected pairwise distance between two uniform empirical distributions, which coincides with the Wasserstein distance only under very restrictive conditions. Calling this an 'approximation' of the Wasserstein distance is misleading and may lead to incorrect completeness conclusions. The section should either use a proper optimal transport solver or relabel the metric as an average pairwise distance and state what it does and does not measure.","section":"Section 3.3.2, Eq. for W(C,A)"},{"comment":"The mapping from decision-boundary distance to a probability score is not correct as written. For a linear classifier with logit z = w^T x + b, the logistic probability is sigma(z), whereas the manuscript defines D = z / ||w|| and then applies sigma(D). Unless ||w|| = 1, sigma(D) is not the model's probability and cannot be interpreted as an entailment probability. This matters because the calibrated human-judgment mapping in Section 5 is applied to machine scores; using a wrongly scaled score changes the calibration curve and the resulting thresholds. The formula should be corrected to use sigma(z), or the distance should be defined and calibrated explicitly as a separate score.","section":"Section 3.2.2"}],"minor_comments":[{"comment":"In the formula for the average pairwise distance, the argument d(ci, sj) should be d(ci, aj); as printed, it refers to an undefined variable sj.","section":"Section 3.3.2"},{"comment":"The quantile notation 'Quantile(...; ceil((n+1)(1-alpha))/(n+1))' is ambiguous; the manuscript should specify the empirical quantile level, e.g., the level is ceil((n+1)(1-alpha))/(n+1) and the quantile is taken over the calibration non-conformity scores.","section":"Section 5.2"},{"comment":"Several figures are referenced but their content is not described in enough detail. Figure 1 has no axis labels, and Figures 4-7 have minimal or no captions describing the data source, the evaluation metric, or the plotted quantity. If these are illustrative examples, that should be stated explicitly.","section":"Figures 1, 4-7"},{"comment":"The reference list includes entries that are not cited in the text (e.g., Khashabi et al. 2021, Zhao et al. 2023), and at least one in-text citation style issue appears in the reference for Yang et al. (2018), where 'Yang, Z. Qi, P.' should be 'Yang, Z., Qi, P., ...'.","section":"References"},{"comment":"The term 'Monotonic XGBoost' is used without explanation; the authors should clarify what monotonic constraints are being imposed and provide a citation or definition.","section":"Section 5.1"},{"comment":"The test-generation step uses an LLM to create queries, but the manuscript does not discuss how the choice of this LLM, its prompt, or its potential biases affect the coverage and representativeness of the generated test set.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like an industry white paper than a research article. The proposed framework is a synthesis of known techniques, and the novelty is limited to the integration and the banking context. The paper relies heavily on self-citations and makes broad reliability claims without empirical support. If the journal accepts position papers or framework overviews, the paper could be acceptable after major revision; otherwise, the lack of any validation study is a serious obstacle. I recommend major revision with the expectation that the authors either add a substantial case study or substantially weaken the claims and reframe the contribution as a proposal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper up front: it is a well-organized overview of a proposed evaluation framework, not a validated method. The HCAT framework combines stratified query generation, embedding-based metrics, NLI for groundedness, and a two-stage calibration with conformal prediction. The synthesis is thoughtful and the target audience—model risk managers in banking—is underserved. The paper reads like a senior practitioner's checklist, and the structure is genuinely clear.\n\nThat said, the central claim that HCAT is a 'practical and reliable solution' has no empirical backing. There are no experiments, no code, no baselines, and no human-labeled data. For a paper whose entire selling point is calibration to human judgment, that is a serious gap. I agree with the reader's CONDITIONAL verdict.\n\nThe stress test about exchangeability lands. Section 5.2 correctly notes that split conformal prediction assumes exchangeability, but the calibration set is built from stratified sampling of the document collection, not from the deployment query distribution. Real production queries will shift in topic, phrasing, and difficulty. The conformal coverage guarantee does not survive that shift, and the paper does not acknowledge it. That is load-bearing, because the reliability claim is exactly what the framework promises. The limitations section mentions topic modeling quality and human sample size but stays silent on distribution shift.\n\nThere are also two metric-level blemishes that are minor but real. Section 3.2.2 maps decision-boundary distance D to probability via sigma(D), but the model's logit is z = ||w||D, so the correct mapping is sigma(z). Section 3.3.2 calls the average pairwise embedding distance the Wasserstein distance, but that average is only an upper bound (independent coupling), not the optimal transport cost. Neither is fatal, but both need fixing before this could be taken seriously.\n\nWho is this for? Practitioners in regulated industries wanting a structured evaluation checklist. Researchers looking for empirical advances will be disappointed. As a serious referee, I would not send this to peer review in its current form—the lack of any proof-of-concept and the overclaiming in the abstract are disqualifying for a research venue. If the authors add a real case study with human labels and correct the metric issues, it could become a useful applied paper.","headline":"A coherent framework overview for RAG evaluation in banking, but it overclaims 'reliable' without empirical support and has a load-bearing exchangeability gap in its conformal step.","tokens_in":14672,"tokens_out":2493,"would_cite":false,"duration_ms":26050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The HCAT framework aligns automated evaluations of RAG-based language models with human judgment through stratified test generation, embedding-based metrics, and two-stage calibration.","keywords":["Generative Language Models","Retrieval-Augmented Generation","Model Validation","Human-Calibrated Testing","Automated Test Generation","Embedding-Based Metrics","Conformal Prediction","Robustness Testing"],"falsifier":"A concrete test would be to calibrate HCAT on one document collection and apply it to a different collection sharing overlapping topics; if the calibrated thresholds yield systematically misaligned human agreement rates—for instance, high machine groundedness scores on answers that humans judge as hallucinated—the generalization claim would be falsified. An adversarial benchmark that separates semantic similarity from factual support, such as paraphrased false statements that are semantically close to the context, would similarly show whether groundedness tracks truth rather than lexical proximity.","tokens_in":13671,"feed_emoji":"🧪","tokens_out":5945,"duration_ms":47769,"temperature":0.7,"pith_summary":"This paper tries to establish that a Human-Calibrated Automated Testing (HCAT) framework can make evaluation of retrieval-augmented generation (RAG) systems both scalable and aligned with human judgment. It targets high-stakes domains like banking, where generative language models must pass rigorous validation before deployment. The framework generates diverse test queries via topic-stratified sampling, scores responses with transparent embedding-based metrics for relevance, groundedness, completeness, and risk, and then calibrates those scores against human labels using probability calibration and conformal prediction. If the framework works, automated evaluation could replace opaque LLM judges and exhaustive manual review, giving regulated industries a practical path to trustworthy model validation.","feed_headline":"Framework calibrates AI answer scores against human judgment","feed_subtitle":"For high-stakes fields like banking, HCAT makes automated evaluation transparent, scalable, and regulator-friendly.","key_machinery":"The load-bearing mechanism is the double-calibration of embedding-derived scores against human labels. In Stage 1, probability calibration (logistic, isotonic, or ordinal regression) converts raw machine scores such as cosine similarity or entailment logits into probabilities that match observed human judgments. In Stage 2, split conformal prediction uses a separate hold-out set to compute a calibrated quantile of non-conformity scores, yielding prediction sets (single class, both classes, or empty) at a user-specified confidence level. This two-stage process is what transforms abstract similarity numbers into decision-ready evaluations with quantified uncertainty.","core_discovery":"The central claim is that the open-ended, subjective nature of generative outputs can be tamed by exploiting the bounded structure of RAG systems, where responses are grounded in a known document collection. HCAT builds an automated test suite by embedding documents, clustering them into topic strata, and sampling queries within each stratum so that all knowledge areas are covered. Evaluation uses sentence-level cosine similarity between embeddings—from contrastively trained models and NLI models—to compute context relevancy, groundedness, completeness, and answer relevancy, with NLI entailment distance and Wasserstein distance as supplementary checks. A two-stage calibration pipeline then maps these machine scores to human-judgment probabilities and attaches confidence-aware prediction sets. The paper argues that this layered design delivers explainability, regulatory acceptance, and targeted weakness identification at scale.","pith_inferences":["The paper assumes that one calibration function learned on a given topic mix transfers to new topics; an empirical study of cross-topic calibration transfer would be a natural next step, since the paper provides no such evidence.","Because the metrics inherit the properties of the underlying embedding and NLI models, a skeptical reader should probe whether calibrated scores stay aligned for demographic subgroups or adversarial paraphrases.","The Wasserstein completeness approximation, computed as an averaged pairwise distance, is a simplification of true optimal transport; exact solvers could change completeness rankings on long, multi-topic documents.","The framework's logic could extend to non-RAG generative models by treating any retrieved or provided context as a grounding reference, though the paper does not explore this generalization."],"forward_implications":["RAG systems can be tested comprehensively across all topics in their document collection without hand-crafting test queries.","Banks and other regulated users can set evaluation thresholds that demonstrably track human judgments, supporting regulatory acceptance.","Conformal prediction sets flag outputs that need human review, enabling a human-in-the-loop monitoring process.","Marginal and bivariate weakness analysis identifies specific topics or query types that underperform, guiding targeted model improvement."],"supporting_citations":[{"why":"Supplies the token-level semantic similarity approach that the paper extends to sentence-level metrics.","marker":"Zhang et al., 2020"},{"why":"Defines retrieval-augmented generation, the system type the framework evaluates.","marker":"Lewis et al., 2020"},{"why":"Provides the foundational conformal prediction framework used in Stage 2.","marker":"Vovk et al., 2005"},{"why":"Offers the split conformal prediction methodology that the paper adopts.","marker":"Angelopoulos and Bates, 2023"},{"why":"Provides natural language inference grounding for the groundedness metric.","marker":"MacCartney, 2009"},{"why":"Supplies the topic modeling method used to define sampling strata.","marker":"Grootendorst, 2022"},{"why":"Provides SimCSE contrastively trained embeddings for similarity scoring.","marker":"Gao et al., 2021"},{"why":"Provides Sentence-BERT sentence embeddings used throughout the metrics.","marker":"Reimers and Gurevych, 2019"},{"why":"Motivates the banking model validation standards that frame the paper's requirements.","marker":"Sudjianto and Zhang, 2024"}],"fun_headline_variants":["Human-calibrated automated testing validates AI answers in banking","HCAT framework aligns machine scores with human judgment for RAG","Automated AI testing gets human calibration for high-stakes domains","Calibrated automated testing for trustworthy generative AI in finance","Explainable AI evaluation framework for banking RAG systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that embedding-based similarity scores and NLI entailment probabilities, after calibration against human labels, reliably measure quality dimensions such as relevance, groundedness, and completeness in a way that generalizes across topics and query types.","fun_headline_variants_meta":{"raw":{"variants":["Human-calibrated automated testing validates AI answers in banking","HCAT framework aligns machine scores with human judgment for RAG","Automated AI testing gets human calibration for high-stakes domains","Calibrated automated testing for trustworthy generative AI in finance","Explainable AI evaluation framework for banking RAG systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1352,"prompt_tokens":908,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":524,"tokens_out":444,"duration_ms":4885,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:08:56.761698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to calibrate HCAT on one document collection and apply it to a different collection sharing overlapping topics; if the calibrated thresholds yield systematically misaligned human agreement rates—for instance, high machine groundedness scores on answers that humans judge as hallucinated—the generalization claim would be falsified. An adversarial benchmark that separates semantic similarity from factual support, such as paraphrased false statements that are semantically close to the context, would similarly show whether groundedness tracks truth rather than lexical proximity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the token-level semantic similarity approach that the paper extends to sentence-level metrics."},{"cited_title":", and Kiela, D","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the system type the framework evaluates."},{"cited_title":"and Shafer, G","cited_arxiv_id":null,"evidence_quote":"Provides the foundational conformal prediction framework used in Stage 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers the split conformal prediction methodology that the paper adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides natural language inference grounding for the groundedness metric."},{"cited_title":"and Chen, D","cited_arxiv_id":null,"evidence_quote":"Provides SimCSE contrastively trained embeddings for similarity scoring."},{"cited_title":"and Gurevych, I","cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT sentence embeddings used throughout the metrics."},{"cited_title":"and Zhang, A","cited_arxiv_id":null,"evidence_quote":"Motivates the banking model validation standards that frame the paper's requirements."}],"review_version":1}