{"id":"3731a422-c6c9-4243-9aac-0bfbbf54058c","arxiv_id":"2608.02444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Partial LLM-agent evaluations can support a promote/reject decision after 15–25% of tasks on some benchmarks, but not others; reports should state the decision rule and unresolved count.","lead":"ParEvalLayer is a decision layer that reads partial outcomes from LLM-agent benchmark runs and says whether one agent is already provably better, not better, or still undecided. Replaying five public benchmarks shows some suites can be decided after 15–25% of tasks while others need almost all tasks, so a bare partial score is not a safe report.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Primary results assume uniform random task order; coverage rule only checks group counts, not within-group representativeness, so Table II budgets may not transfer to fixed or cost-biased live orders.","rationale":"The reader's verdict is CONDITIONAL and the stated weakest assumption is the random task-order premise. I agree. The paper's replay methodology averages over random permutations, which yields a distribution of subsets. Sufficiency is defined as aggregate error rates over these random cases. A live harness does not randomize task order; it typically has a fixed schedule (e.g., by repository, by difficulty, or by cost). The coverage rule ensures each stratum is represented in the observed subset, but it does not ensure the tasks within a stratum are representative. If the observed subset is biased toward easy or cheap tasks, the bootstrap tails can be artificially narrow, and the decision may not match the completed record. The paper's tau-bench cost-aware check is evidence that order matters, but because that check uses a forced decision rule that skips the coverage safeguard, it does not directly test the primary policy's robustness. The omission is that the primary policy is never replayed under a deterministic non-random order. This is not an internal inconsistency: the paper is transparent that it samples random orders, and Section VI states the results do not establish a live stopping guarantee. Still, the headline numbers in Table II are likely to be read as 'these benchmarks are decidable at 15-25%' without the random-order caveat. A single replay under a realistic biased order would settle whether the concern is only theoretical. If the primary policy remains within targets, the central claim is robust; if not, the budgets should be qualified. The paper's normative conclusion — that scores alone are insufficient and decision rules should be recorded — remains well supported regardless, so the verdict should remain CONDITIONAL.","tokens_in":12786,"tokens_out":6743,"duration_ms":69623,"concrete_test":"Re-run the Section IV-B replay pipeline on tau-bench with tasks ordered by increasing cost (the cost-aware order) and on one other benchmark (e.g., AppWorld) with tasks ordered by baseline-system success within each stratum (easy-first), while keeping the primary coverage and bootstrap policy active. For each budget, compute the class-conditional false-promotion and false-rejection rates and the minimum sufficient budget. If any rate exceeds the 5% target or the sufficient budget shifts by more than one grid step relative to Table II, the random-order assumption is load-bearing and the Table II budgets should be qualified as random-order-only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Table II, Section V-A) is that AppWorld, OSWorld-Verified, and tau-bench reach the completed-record decision at 15-25% task budget. These numbers are produced by Section IV-B's protocol: 'we sample 500 task orders per ordered comparison' (2,000 for tau-bench). Sufficiency is evaluated against the distribution over uniform random permutations. A live evaluation service typically uses a fixed, deterministic task order, often grouped by repository/domain or ordered by cost/difficulty. For such an order, the observed subset at budget b is not a uniform sample of the shared task set. The coverage rule in Section III-D only enforces that each stratum's required count r_g(b) is met; it does not control which tasks within a stratum appear. If a fixed order presents easy or cheap tasks first within each stratum, the bootstrap tails in Section III-E are computed from a biased subset, so the promote/reject decision can disagree with the completed-record decision at rates above the 5% targets. The paper's own cost-aware forced-evaluation check (Section V-B) demonstrates that task order can produce 100% coverage failure and wrong decisions at 25% budget on tau-bench, but because that variant ignores the coverage rule, it does not establish that the primary policy is robust. The primary policy is never evaluated under a realistic non-random order (e.g., cost-ascending, difficulty-ascending, or grouped-by-stratum) while keeping coverage active. Therefore the minimum sufficient budgets are averages over random orders, not bounds that transfer to any fixed order.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ParEvalLayer, a decision layer for partial LLM-agent evaluations. For a two-system comparison, the layer takes the task outcomes observed so far plus a pre-specified policy (improvement threshold, coverage requirement, uncertainty rule, budget limit) and outputs one of four actions: promote, reject, continue, or abstain. The authors evaluate the layer by replaying completed public benchmark records (SWE-bench Lite/Verified, AppWorld, OSWorld-Verified, tau-bench, and Terminal-Bench) as if the evaluation had stopped at various fractions of the task set or elapsed time, and checking whether promote/reject decisions agree with the completed-record conclusion. Under the primary policy at a 0 percentage-point threshold, the paper reports that AppWorld, OSWorld-Verified, and tau-bench reach sufficiency at 15%, 20%, and 25% task budget, respectively, while SWE-bench Verified requires 90% and SWE-bench Lite has no sufficient budget by 95%. A supplementary cost-aware ordering check on tau-bench shows that non-random cheap-first ordering can produce high coverage failure and wrong decisions, motivating the paper's central recommendation that partial reports should state the decision rule and unresolved-comparison rate rather than only a partial score.","tokens_in":13203,"tokens_out":9983,"duration_ms":110695,"significance":"If the conclusions hold, the paper makes a useful contribution to agent-evaluation reporting. Its strengths are the explicit decision contract (Fig. 2, Eqs. 5-10), the honest replay protocol in which abstentions remain in denominators, the pre-specified policy parameters with no fitted free parameters, and the demonstration that cheap-first task ordering can invalidate a partial comparison. The cross-benchmark variation in sufficient budgets is a valuable negative result: partial-score reporting is not uniformly reliable. The main limitation is that the headline early-budget claims are computed under uniformly random task order, while the paper does not clearly state that randomization is a required part of the policy or evaluate the primary policy under realistic non-random orders with the coverage rule active. This is a scope issue rather than an internal inconsistency, but it affects how the empirical claims can be interpreted and used.","major_comments":[{"comment":"The headline budgets (15-25% for AppWorld, OSWorld-Verified, tau-bench) are produced by sampling task orders per ordered comparison (500 orders, or 2,000 for tau-bench) and averaging over the resulting subsets. Neither the abstract nor the policy specification states that uniform random task order is a required policy element. In a live evaluation with a fixed order (cost-ascending, difficulty-ascending, or grouped by repository/domain), the observed subset at budget b is not a uniform sample of the shared task set. The coverage rule in Eq. (4) checks only per-stratum counts r_g(b), not within-stratum representativeness, and the bootstrap tails in Eqs. (5)-(6) are computed on the observed subset. Section V-B shows that task order can cause 100% coverage failure and wrong decisions on tau-bench, but that check uses forced evaluation without the coverage rule, so it does not establish robu","section":"IV-B / Table II / Abstract, with V-B"}],"minor_comments":[{"comment":"State explicitly that the 500 (2,000 for tau-bench) task orders are sampled uniformly at random, and clarify whether task order is part of the policy itself or only a simulation device. The current wording is ambiguous, especially because Section I lists 'task order' as a fixed policy component.","section":"IV-B"},{"comment":"The coverage-failure target in Eq. (9) is trivially satisfied for the primary policy: Fig. 2 returns CONTINUE or ABSTAIN whenever coverage fails, so the primary policy can never issue a promote/reject before coverage holds. Statements in Section V-A such as 'meets the decision-error and coverage targets' are therefore uninformative for the primary policy. Either define a different coverage-failure quantity (e.g., the rate at which coverage prevents a decision) or state that Eq. (9) is intended only for the non-coverage-aware comparison policies.","section":"III-E and Fig. 2"},{"comment":"The column '0 pp unresolved interval' is not defined in the text. Specify at which budget these intervals are computed and how they are derived from directed-pair variability. The tau-bench interval [7.25, 43.84] is particularly wide and needs a footnote explaining that the 12 directed pairs make this a case-study result, not a population estimate.","section":"Table II"},{"comment":"The sentence 'Paired-normal tests give the same sufficient-budget pattern' is too terse. Since the paired-normal rule is one of the decision-rule variants described in IV-B, give its exact criterion (e.g., t-test or z-test on the paired differences, with what variance assumption) so the comparison is reproducible.","section":"V-C"},{"comment":"The 'small round-budget check' is mentioned but never described or reported. Either state its results or remove the reference, since all headline claims come from the 5-percentage-point sweep.","section":"IV-B"},{"comment":"The phrase 'three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes' should carry the qualifier 'under the stated policy with randomized task order' to avoid over-generalization.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid reporting-layer contribution with an honest replay protocol. I have no concern about circularity: the thresholds are pre-specified and no equation reduces to a fitted parameter. The main issue is that the headline early-budget claims are tied to uniformly random task order, while the paper does not clearly communicate this or test the primary policy under the non-random orders that live evaluation services commonly use. If the authors either add the missing non-random-order experiments with coverage active or explicitly scope all claims to randomized order, the paper should be publishable. A data/code availability statement would also strengthen reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ParEvalLayer is a genuine, if modest, addition to the LLM-evaluation toolkit. The core idea—that a partial agent evaluation should output a decision record (promote/reject/continue/abstain) with coverage and bootstrap tails, rather than a raw partial score—is new relative to the subset-selection and sequential-testing work it cites. The replay protocol is honestly designed: abstentions stay in denominators, the coverage rule is simple and explicit, and the tau-bench cost-aware check shows that ordering effects can wreck naive early stopping. The cross-benchmark variation (15% to 25% for three benchmarks, 90% for SWE-bench Verified, none by 95% for SWE-bench Lite) makes the reporting recommendation concrete.\n\nThe soft spot is the one the stress-test flags. All headline budgets come from averaging over uniform random task orders—the paper samples 500 (or 2,000) random permutations per comparison. Real evaluation harnesses use fixed orders, often grouped by domain or sorted by cost. The coverage rule only checks that each stratum's required count is met; it doesn't control which tasks within a stratum appear. So if a fixed order presents easy or cheap tasks first within each group, the bootstrap tails are computed from a biased subset, and the promote/reject decision can diverge from the completed-record decision more often than the 5% targets allow. The paper demonstrates this failure mode in the cost-aware forced-evaluation variant on tau-bench, but that variant ignores the coverage rule, so it doesn't show whether the primary policy is robust. The primary policy is never tested under a realistic non-random order with coverage active.\n\nThat said, this is a limitation rather than a fatal flaw. The authors are unusually honest: the limitations section says the experiments don't establish a live stopping guarantee, and the budgets are meaningful only through the recorded policy. My main suggestions for revision are to add a sensitivity check under cost-ascending or grouped-by-domain task orders with coverage active, and to release the replay code, since no code is provided. The paper is well-scoped, clearly written, and the central claim—partial reports need decisions, not just scores—holds up. It deserves a serious referee and would be a useful contribution to the agent-evaluation methodology literature.","headline":"A well-scoped, honest paper that introduces a decision record for partial agent evaluations; the headline 15–25% budgets are conditional on random task order, but the core reporting idea is sound and deserves peer review.","tokens_in":13648,"tokens_out":3276,"would_cite":true,"duration_ms":33780,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Partial agent evaluations should report decisions, not scores, because the evidence needed to decide varies from under a fifth of tasks to more than the full run.","keywords":["LLM agents","partial evaluation","early stopping","benchmark reporting","decision records","coverage rule","bootstrap uncertainty","abstention"],"falsifier":"Take one of the benchmarks that the paper reports as sufficient at 20% of the task budget, run the evaluation with a fixed task order that places the most difficult or most cost-heavy tasks last, and check whether a promote/reject decision made at 20% flips when all tasks are observed. If a flipped decision is found, the early-sufficience claim is falsified for that task order.","tokens_in":12720,"feed_emoji":"⏱️","tokens_out":4747,"duration_ms":46157,"temperature":0.7,"pith_summary":"This paper argues that a partial LLM-agent evaluation is only meaningful if it records whether the evidence supports a decision, not just a raw score. It introduces ParEvalLayer, a decision layer that takes paired task outcomes and a pre-committed comparison policy, then outputs one of four actions: promote, reject, continue, or abstain. Replaying completed public benchmarks as if runs had stopped early, the paper finds that three benchmarks reach the same conclusion as the full evaluation after 15% to 25% of tasks, while another requires 90% and one never resolves within 95%. This variation is the point: a score alone cannot tell a reader whether a partial run's conclusion would survive the missing tasks. The paper proposes that evaluation reports state the decision rule, coverage status, and how many comparisons remain unresolved.","feed_headline":"Stop agent evals early; report decisions, not scores","feed_subtitle":"Three benchmarks reach the full-run verdict after 15-25% of tasks; some never resolve.","key_machinery":"The load-bearing object is the decision record produced by ParEvalLayer. It has four actions—promote, reject, continue, abstain—and is generated by a routine that first enforces task-group coverage (every nonempty benchmark group must appear at least once and in proportion to its size) and then applies a bootstrap tail check on the paired differences to decide whether the observed margin is decisive. The sufficiency of a budget is judged by three aggregate targets: conditional false-promotion and false-rejection rates below 5%, coverage failures below 5%, and unresolved comparisons below 25%. These targets make it impossible for a rule to look accurate by deciding only easy pairs.","core_discovery":"The central discovery is that partial-agent-evaluation reporting is a decision problem, not a score-estimation problem. ParEvalLayer converts a partial run into a decision record by first checking that the observed tasks cover all required benchmark groups (repository, difficulty split, application, domain), then applying a bootstrap tail test to see whether the observed paired margin is far enough from the improvement threshold. Promotes and rejects are only returned when coverage holds and the bootstrap tail is below 5%; otherwise the layer continues or abstains. Replaying completed public records, the paper shows that with a 0 percentage-point threshold, three benchmarks support the same","pith_inferences":["The paper's early-sufficiency results are conditioned on uniform random task order; a natural extension is to test the decision layer on cost-ordered or adversarial task orders, where the paper's own tau-bench check suggests failures can be severe.","The 15-25% early-decision benchmarks may have large clusters of nearly homogeneous tasks, so an adaptive, learned task-ordering policy could push even the high-budget benchmarks below 90%; that is an untested consequence of the coverage rule.","One could turn the decision record into a live protocol: a benchmark service fixes the policy, then runs tasks in random order and stops at the first budget where the record says promote or reject, with the unresolved rate as a retrospective quality measure.","The abstention target (25%) has no statistical grounding; tightening or loosening it changes which budgets count as sufficient, so reporting suites should treat the target as a policy choice, not a universal constant."],"forward_implications":["Evaluation services can stop runs early and publish a comparison as soon as the decision record says promote or reject, saving wall-clock time and API budget without claiming more evidence than the policy allows.","Benchmark reports that include only a score become visibly incomplete: readers can ask for the decision rule, coverage status, and unresolved-comparison count before accepting a partial conclusion.","The decision layer can be attached to any existing evaluation pipeline that writes task-level paired outcomes, without changing task execution or scoring.","Budget sufficiency numbers are only meaningful with the policy: thresholds, task order, coverage rule, and abstention target all shift the minimum budget, so a task fraction alone is not a property of a benchmark.","In live use, a valid sequential or confidence-sequence boundary would be needed for repeated looks; the paper's bootstrap tail supports the replay comparisons but not time-uniform guarantees."],"fun_headline_variants":["Decide early: ParEvalLayer cuts agent eval time","Partial evals: when to trust the verdict","Agent evals: 3 benchmarks decide at 15-25% tasks","ParEvalLayer: know when to stop agent evaluation","Partial scores mislead; ParEvalLayer reports verdicts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The early-sufficiency results assume that the tasks observed in a partial run are a uniformly random sample of the full task set; if live task order is fixed or cost-ordered, the observed subset is not random and the same decisions may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Decide early: ParEvalLayer cuts agent eval time","Partial evals: when to trust the verdict","Agent evals: 3 benchmarks decide at 15-25% tasks","ParEvalLayer: know when to stop agent evaluation","Partial scores mislead; ParEvalLayer reports verdicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2686,"prompt_tokens":759,"completion_tokens":1927,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1842}},"tokens_in":503,"tokens_out":1927,"duration_ms":14311,"temperature":1.0,"reasoning_tokens":1842,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:13:56.310237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the benchmarks that the paper reports as sufficient at 20% of the task budget, run the evaluation with a fixed task order that places the most difficult or most cost-heavy tasks last, and check whether a promote/reject decision made at 20% flips when all tasks are observed. If a flipped decision is found, the early-sufficience claim is falsified for that task order.","supporting_citations":[],"review_version":1}