{"id":"440249b1-9dd6-45ab-89f4-7536175d65a6","arxiv_id":"2608.12970","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"REAG and a confidence-calibrated cascade generate context-aware test oracles for LLM-based software and produce statistically controlled verdict reliability, demonstrated on a production nutrition advisory app.","lead":"This paper introduces a framework for automatically generating and checking acceptance tests for software powered by large language models, using the software's own requirements and user profiles to decide what correct behavior looks like. It adds a confidence filter that escalates uncertain verdicts to stronger models or abstains, and an industrial trial on a nutrition app reports 98.8% verdict accuracy and a 31.7% cost saving.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's cascade cost omits mandatory J1/J2 calls; including them per §3.4.1 reverses the claimed 31.7% cost-efficiency advantage.","rationale":"The paper makes three connected claims: REAG produces useful oracles, the cascade accurately filters unreliable verdicts, and the cascade is more cost-efficient than single judges. The reader's weakest assumption concerns exchangeability of synthetic personas with deployment data. That is a real limitation and the authors disclose it, but it is a standard caveat that does not by itself overturn the empirical results. The cost-accounting flaw is more load-bearing because it is internal to the paper's own algorithm and arithmetic. Section 3.4.1 states that every item begins at the lowest-cost judge and escalates only on insufficient confidence, but Table 3 charges only the judge that issued the final verdict. Recomputing with mandatory J1 and J2 calls turns the reported 31.7% cost advantage over Gemini-2.5-Flash into a small cost disadvantage at the headline operating points. This concern is concrete, checkable by arithmetic, and independent of distributional assumptions; it directly affects the 'industrial viability' claim in the abstract and conclusion. I still credit the paper for a transparent calibration/evaluation split, disclosed failure analysis, and a clear per-tier conformal calibration procedure; those are genuine strengths and independent support. However, the central cost-efficiency number needs correction. The appropriate verdict is conditional: the paper should be accepted for publication only after the authors recompute cost-efficiency under the actual cascade call pattern and restate the RQ3 conclusion accordingly.","tokens_in":21133,"tokens_out":18392,"duration_ms":187926,"concrete_test":"Recompute every row of Table 3 using the cascade's actual call pattern: C(α) = 100×cost(J1) + N_J2(α)×cost(J2) + N_J3(α)×cost(J3), where N_J2 is the number of items not accepted by J1 and N_J3 is the number escalated past J2, with costs 1, 3, and 4.6 per item as stated in §4.7. For α=0.14, C = 100 + 100×3 + 27.9×4.6 = 528.3, CPP = 5.35 vs J3 baseline 5.11; for α=0.16, C = 100 + 100×3 + 17.2×4.6 = 479.1, CPP = 5.27. If the published table used final-tier-only cost, the '31.7% cost reduction' claim fails; if a different cost definition was intended, it must be stated explicitly and all baselines recomputed under the same definition.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.4.1 specifies that every verdict starts at J1 and escalates only when confidence falls below the tier threshold, so every item incurs J1 cost, and every item escalated to J3 also incurs J2 cost. Table 3's 'Total Cost' for the cascade is instead computed as final-deciding-tier cost only: at α=0.14 it uses 72.1×3 + 27.9×4.6 = 344.6, charging nothing for J1 on any item and nothing for J2 on the 27.9% of items escalated to J3. Under the stated algorithm, actual cost per 100 units at α=0.14 is 100×1 + 100×3 + 27.9×4.6 = 528.3, giving CPP = 528.3/98.8 = 5.35, which is worse than J3 alone (CPP = 460/90 = 5.11). At α=0.16, actual cost is 100 + 300 + 17.2×4.6 = 479.1, CPP = 5.27, again above J3's 5.11. Thus the headline '31.7% more cost-efficient than the strongest single judge' is not an exchangeability caveat but an arithmetic inconsistency with the paper's own cascade algorithm. The same omission also makes the cascade look cheaper than the J2 baseline, which it is not once J1 calls are counted. The paper's cost-efficiency conclusion therefore needs a corrected cost model before it can support the central viability claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an automated acceptance-testing framework for LLM-based software (LBS), with two main contributions. First, Requirements-Augmented Generation (REAG) interprets user intentions by retrieving software requirements, personas, and domain knowledge via adaptive RAG and self-reasoning, producing executable test oracles. Second, a confidence-calibrated cascade of LLM judges uses simulated expert agreement and conformal risk control to accept, escalate, or abstain on verdicts, with the abstention mechanism acting as a quality gate. The evaluation is an industrial case study on a nutrition advisory application with 346 scenarios, a 246-item calibration set, and a 100-item evaluation set. The paper reports that REAG achieves 3.91/5 oracle quality, that the cascade reaches 98.8% selective agreement at alpha=0.14 with 14% abstention, and that it offers a 31.7% cost-efficiency improvement over the strongest single judge.","tokens_in":21439,"tokens_out":6936,"duration_ms":67848,"significance":"If the reported results held, the paper would make a timely and useful contribution: it applies conformal risk control to acceptance testing for LLM-based software, combines oracle generation with reliability-aware judging, and provides an industrial case study. The empirical design is stronger than much of the LLM-as-judge literature: calibration and evaluation sets are separated, calibration labels were frozen before evaluation, expert agreement is reported (Fleiss' kappa 0.76), failure analysis is explicit, and the replication package is public. The framework's core idea is credible and the paper's framing of limitations is unusually candid. However, the cost-efficiency claim in RQ3 is inconsistent with the cascade algorithm described in Section 3.4.1, and the headline selective-agreement number is based on a small accepted set with a mix of selective and full-set metrics. These issues affect the central viability claim and require correction.","major_comments":[{"comment":"The cost model in Table 3 charges only the final deciding tier. Under the cascade algorithm in Section 3.4.1, every item is first processed by J1, and every item escalated to a higher tier also incurs the cost of all lower tiers. At alpha=0.14, the judge distribution is 0/72.1/27.9, so the actual cost per 100 units is 100*1 + 100*3 + 27.9*4.6 = 528.3, not 344.6. The corrected CPP is 528.3/98.8, approximately 5.35, which is worse than the J3-alone baseline (460/90, approximately 5.11). At alpha=0.16, the corrected cost is 100 + 300 + 17.2*4.6 = 479.1, giving CPP approximately 5.27, again above the J3 baseline. Therefore the claimed 31.7% cost-efficiency advantage over Gemini-2.5-Flash, and the statement that the cascade is more cost-efficient than the J2 baseline, are not supported by the paper's own algorithm. RQ3, the abstract, and the conclusion must be recomputed, or the table must explicitly state that it reports deciding-tier cost only, with a clear justification for why that is the relevant cost metric.","section":"Section 3.4.1, Table 3, Section 4.7 (RQ3 Answer)"},{"comment":"The headline 98.8% selective agreement at alpha=0.14 is computed on the accepted subset, which is 86% of the 100 evaluation items, i.e., approximately 86 verdicts, and it corresponds to a single disagreement. This number is reported as a precise percentage and compared directly with full-set accuracies of single judges (90% and 83%), mixing selective agreement with unconditional accuracy. Please report the number of accepted items in the denominator, provide a binomial confidence interval for the selective agreement, and compare all systems on the same basis, either by applying the same acceptance rule to the baselines or by reporting both selective and full-set metrics for every system.","section":"Section 4.6, Figure 4"},{"comment":"The statement that the cascade 'improves average oracle quality from 3.91 to 4.30' describes a property of the filtered retained set, not a causal improvement to oracle generation. Because the filter is correlated with the rated quality, the retained-set mean is higher by construction. The paper should state this as the selective quality of the accepted verdicts rather than as an 'improvement' in oracle quality. As written, the abstract and conclusion overstate the result.","section":"Section 4.6, Figure 5, Abstract, Conclusion"}],"minor_comments":[{"comment":"Reporting a single Fleiss' kappa of 0.76 'across D3, D4, and D5' is ambiguous because those data categories involve different annotation tasks (Likert-style ratings, binary verdicts, and interviews); please report per-task kappa values.","section":"Section 4.2.2"},{"comment":"The notation '88* is the last observable point' is unexplained; please define the asterisk and state why only 88 of 100 evaluation points are observable at that setting.","section":"Figure 4"},{"comment":"The definition of the upper confidence bound in Eq. (6) is not transparent; specify the Clopper-Pearson construction, the confidence level, and the direction of the supremum so that the formula can be verified independently.","section":"Section 3.4, Eq. (6)"},{"comment":"The symbol K is used for the number of in-context examples in the prompt and also appears in the adaptive top-k retrieval discussion; rename one of these to avoid confusion.","section":"Section 3.4.1"},{"comment":"Reference [54] has a DOI that currently resolves to a preprint server rather than the cited ACM TOSEM version, and reference [60] contains duplicated author names; both should be corrected.","section":"References"},{"comment":"The 82% qualified/marginal rate uses a per-dimension flag rule, while the composite 3.91/5 average uses a different threshold; the text explains this, but a small table with both definitions would make it easier to verify.","section":"Section 4.5 (RQ1)"}],"recommendation":"major_revision","confidential_remarks":"The cost-accounting error in RQ3 is the decisive issue; it is load-bearing for the central viability claim, but it is fixable by recomputing the cost model and rewording the affected claims. I am therefore not recommending rejection. Please ask the authors to redo Table 3, RQ3, the abstract, and the conclusion with the corrected costs, and to align the selective-agreement reporting with standard confidence intervals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, it's a serious industrial case study of a real gap: acceptance testing for LLM-based software, where traditional oracles don't work. The framework—REAG for generating test oracles from requirements, personas, and domain knowledge, plus a three-tier LLM cascade with conformal calibration and abstention—is sensible, and the evaluation is more careful than most SE case studies: separate calibration (246) and evaluation (100) sets, frozen labels, Fleiss' κ=0.76, and explicit failure analysis. Second, the headline cost-efficiency claim is wrong by the paper's own algorithm. Section 3.4.1 says every item starts at J1 and escalates only when confidence is low, so every item incurs J1 cost and every item reaching J3 also incurs J2 cost. Table 3's Total Cost for the cascade is computed as if only the final tier is charged (at α=0.14: 72.1×3 + 27.9×4.6 = 344.6). Charging 100×1 + 100×3 + 27.9×4.6 = 528.3 gives CPP = 5.35, worse than J3 alone (5.11). So the claimed 31.7% cost-efficiency advantage reverses, and the cascade also isn't cheaper than J2. That's not an exchangeability caveat; it's an arithmetic inconsistency with the stated algorithm, and it undercuts the RQ3 viability conclusion.\n\nWhat the paper does well: the reframing of RAG as behavioural specification is genuinely useful, and the cascade's abstention as a quality gate is a practical idea. The reliability analysis is transparent—the authors disclose that per-tier conformal guarantees do not compose under adaptive routing, that synthetic persona generation may shift the distribution, and that annotators came from the development team. The 98.8% selective agreement on the accepted set with 14% abstention is a real number, but it's measured on 100 items; treat it as indicative, not proof. The other soft spots are minor relative to the cost problem: the per-tier guarantees are formally separate, temperature scaling relaxes them, and the full dataset is proprietary.\n\nBottom line: anyone working on QA for LLM-based systems should read this and cite it (with the cost caveat). It deserves peer review—the idea and the evaluation are solid enough to warrant referee time—but the cost model must be fixed before the paper can support its central viability claim. I would send it out with a request for a corrected cost analysis.","headline":"Worth reading and worth sending out, but the 31.7% cost-efficiency claim is refuted by the paper's own cascade algorithm.","tokens_in":697,"tokens_out":1485,"would_cite":true,"duration_ms":31802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Acceptance testing of LLM-based software can be made trustworthy by generating oracles from requirements and personas and calibrating a cascade of LLM judges, reaching 98.8% accuracy and 31.7% lower cost in an industrial case.","keywords":["LLM-based software","Acceptance testing","Test oracle","Requirements engineering","Retrieval-augmented generation","Conformal risk control","LLM-as-a-judge","Cascade judgment"],"falsifier":"Collect a fresh set of real, non-synthetic user queries from the same application, run the cascade with the paper's calibrated thresholds, and compare its accepted verdicts against expert labels; if the observed disagreement rate exceeds the calibrated alpha, for instance 0.14, by more than sampling error, the exchangeability assumption fails and the finite-sample guarantee does not hold outside the calibration distribution.","tokens_in":20897,"feed_emoji":"🧪","tokens_out":11252,"duration_ms":91133,"temperature":0.7,"pith_summary":"LLM-based software is intent-driven: the same query needs different answers for a diabetic user, an athlete, or a pregnant user, so a fixed expected answer cannot serve as an acceptance-test oracle. This paper claims that the missing oracle can be reconstructed from the software's own requirements, personas, and domain knowledge, and that the reliability of the resulting verdicts can be calibrated statistically. It introduces REAG, which retrieves these software artifacts and reasons over them to produce executable pass/fail criteria, and a confidence-calibrated cascade of LLM judges that accepts, escalates, or abstains based on simulated expert agreement, with thresholds set by conformal risk control. In an industrial case study on a production nutrition advisory application with 346 test scenarios and 46 user profiles, the framework produced oracles rated 3.91/5 on average, reaching qualified or marginal quality in 82% of cases, and the cascade achieved 98.8% verdict accuracy while improving retained oracle quality to 4.30 and cutting cost by 31.7% versus the strongest single judge. The stakes are practical: if this holds, stochastic, personalised LLM systems can be acceptance-tested automatically rather than by human inspection alone.","feed_headline":"Cascade judge makes LLM acceptance tests 98.8% accurate","feed_subtitle":"Oracles written from user needs and personas, with a self-calibrating judge that abstains when unsure, cut cost a third.","key_machinery":"The pipeline's load-bearing components are REAG and the cascade judge. REAG uses ICRALM scoring, the average log-probability of the user intention given a candidate document, to rank software artifacts; an adaptive top-k rule that cuts off at the largest similarity gap; and self-reasoning steps that check relevance, cite evidence, and synthesise an interpreted intention. The interpreted intention is then rendered into an oracle with a TestObjective, an ExpectedResultsDescription, and explicit PassFailCriteria. The cascade is a tiered set of LLM judges from different model families whose confidence is estimated by treating N simulated annotators, each conditioned on K in-context examples, and averaging their label probabilities. Calibrated thresholds are chosen by fixed-sequence testing with a binomial upper confidence bound on disagreement risk, and if even the strongest judge falls below threshold the criterion is abstained and sent to human review. This combination turns an uncalibrated 'does the response look right' judgment into a decision rule with a stated risk bound on accepted verdicts.","core_discovery":"The central claim is that acceptance testing for LLM-based software should be reframed as requirements-grounded intent reconstruction rather than output comparison. The paper argues that user persona is a first-class input: a correct response is defined by who is asking and under what constraints, so any oracle that ignores the asker is underspecified. REAG operationalises this by ranking and retrieving requirements, domain knowledge, and persona descriptions, dynamically choosing how many artifacts to retrieve, and generating an oracle as a structured set of pass/fail criteria. The cascade then treats the verdict as a statistical decision: each judge estimates confidence through simulated expert agreement, low-confidence verdicts escalate to a stronger judge, and if no judge is confident enough the system abstains rather than emitting an unreliable verdict. Conformal risk control calibrates the confidence thresholds so that, on the calibration distribution, the disagreement rate of accepted verdicts stays below a chosen risk level with finite-sample guarantees; the paper reports that empirical human agreement met or exceeded the target across all tested risk settings.","pith_inferences":["If retrieval precision bounds oracle quality as the paper argues, then corpus quality, such as partitioning technical from user-facing requirements, becomes a testable upstream intervention; reorganising artifacts could be measured directly against the 18% unqualified-oracle rate.","Because the calibration profiles are partly synthetic, an obvious next experiment is to calibrate only on real production queries and compare accepted-verdict disagreement rates with the reported 98.8%.","The cascade's abstain-on-doubt design generalises beyond acceptance testing to any LLM output validation task where a small expert-labelled calibration set is affordable, such as content safety or medical triage checks.","The reported 82% qualified-oracle rate suggests that current RAG retrieval will be the limiting factor when this framework is ported to domains with sparser requirements documentation, so proxy metrics for artifact retrievability could predict portability."],"forward_implications":["Acceptance testing moves from the service layer to the semantic layer: tests can fail because a response is wrong for this user's health context, not just because an API misbehaved.","The abstention mechanism doubles as a quality filter: at alpha = 0.14 the cascade keeps 86% of criteria for automated verdicts and raises the average quality of the remaining oracles from 3.91 to 4.30.","Reliability guarantees are local: any new deployment must re-run calibration on its own expert-labelled set and obtain its own thresholds; the paper's alpha = 0.14 operating point is not portable.","Cost savings follow from price-capability tiering: routing easy cases to a cheap judge and only escalating hard ones beats always paying for the strongest model, here by 31.7% at equal or better accuracy.","Retrieval precision, not generation quality, sets the ceiling on oracle correctness; improving the organisation of requirements artifacts should directly raise oracle quality."],"supporting_citations":[{"why":"Supplies conformal risk control, the statistical method that calibrates per-judge confidence thresholds with finite-sample risk bounds.","marker":"[4]"},{"why":"Supplies the per-tier conformal calibration procedure and the simulated-annotator confidence estimation used in the cascade.","marker":"[27]"},{"why":"Supplies the adaptive top-k retrieval rule that dynamically selects how many software artifacts to use in REAG.","marker":"[49]"},{"why":"Supplies ICRALM, the relevance-scoring kernel that ranks retrieved documents by query-token log probabilities.","marker":"[43]"},{"why":"Supplies the self-reasoning trajectories that add relevance checking, evidence citing, and analysis to retrieval.","marker":"[60]"},{"why":"Documents overconfidence in LLM-as-a-judge, motivating the cascade's abstention mechanism and cross-family judge selection.","marker":"[50]"},{"why":"Documents correlated errors among LLM judges, motivating selection of judges from different model families.","marker":"[29]"},{"why":"Prior work on acceptance-criteria generation with retrieval-augmented LLMs that REAG extends toward behavioural specification.","marker":"[53]"},{"why":"Provides the four evaluation perspectives, Relevance, Coverage, Correctness, and Coherence, used to score oracle quality.","marker":"[6]"},{"why":"Supplies the cost-per-performance metric used to compare the cascade against single-judge baselines.","marker":"[16]"}],"fun_headline_variants":["Cascade judge with abstention cuts LLM test cost by 31.7%","REAG oracles: 98.8% accurate acceptance tests for LLM software","Cascade judge abstains when unsure, boosts LLM test accuracy to 98.8%","Persona-aware oracles and confidence cascade cut LLM test cost 31.7%","Abstain when unsure: cascade judge delivers 98.8% accurate LLM tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability guarantee assumes that the 246 calibration examples, including user profiles manufactured by a simulated user agent, look like the queries and users the system will actually face in production; if the synthetic profiles do not match real usage, the calibrated thresholds will not keep the verdict error rate at the promised level.","fun_headline_variants_meta":{"raw":{"variants":["Cascade judge with abstention cuts LLM test cost by 31.7%","REAG oracles: 98.8% accurate acceptance tests for LLM software","Cascade judge abstains when unsure, boosts LLM test accuracy to 98.8%","Persona-aware oracles and confidence cascade cut LLM test cost 31.7%","Abstain when unsure: cascade judge delivers 98.8% accurate LLM tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4465,"prompt_tokens":1050,"completion_tokens":3415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":3310}},"tokens_in":666,"tokens_out":3415,"duration_ms":22211,"temperature":1.0,"reasoning_tokens":3310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:28:55.093072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a fresh set of real, non-synthetic user queries from the same application, run the cascade with the paper's calibrated thresholds, and compare its accepted verdicts against expert labels; if the observed disagreement rate exceeds the calibrated alpha, for instance 0.14, by more than sampling error, the exchangeability assumption fails and the finite-sample guarantee does not hold outside the calibration distribution.","supporting_citations":[{"cited_title":"LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues","cited_arxiv_id":"2604.19464","evidence_quote":"Supplies the self-reasoning trajectories that add relevance checking, evidence citing, and analysis to retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cost-per-performance metric used to compare the cascade against single-judge baselines."}],"review_version":1}