{"id":"71aaa3a8-69c4-4a12-8b1f-a6fdd6ac919f","arxiv_id":"2608.00378","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"CASPER identifies coherent slices of LLM regression tests and ranks regressed slices using execution-log behavioral signals, outperforming random ranking on SWE-bench changes.","lead":"This paper proposes CASPER, a framework that groups regression tests for LLM-based systems into behaviorally coherent slices and then ranks the slices most likely to have regressed after a model or prompt change. It evaluates on SWE-bench verified with four real changes and reports higher slice consistency and better prioritization than clustering and random baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3 gains may come from the sampled ΔFR(s) feature being a direct estimate of the ground-truth label; without a ΔFR-only baseline or ablation, behavioral-signal contribution is unproven.","rationale":"The reader's weakest assumption correctly identifies the same load-bearing issue: the sampled failure-rate change serves both as the classifier's primary feature and as the basis for the ground-truth labels, so the RQ3 comparison against random ranking cannot attribute the gains to behavioral signals. My reading of §2.3.5, §3.4.3, Table 14, and Table 15 reinforces this. The paper's slice identification component has independent support: CASPER-SI achieves high output consistency and generally competitive coherence, with significant differences over clustering baselines. But the ranking evaluation is the core of the claimed contribution, and it lacks the minimal baseline needed to show that the behavioral-signal machinery (failure-prediction model, signal extraction, failure-impact scoring) matters. A direct sampled-ΔFR ranking baseline would settle this cleanly. Since the manuscript as written does not provide that evidence, the REJECT verdict stands; the concern does not move the verdict but explains why the ranking claim is not yet established.","tokens_in":35479,"tokens_out":4821,"duration_ms":44542,"concrete_test":"Add a raw-ΔFR baseline to RQ3: for each dataset and K∈{1..4}, rank slices by the sampled ΔFR(s) computed exactly as in §2.3.5 (descending positive values first, then negative/improved, then zero), with the same CDS/FPS sampling, same class-balancing over 30 repeats, and same Recall@K/AUC metrics. Also retrain the §2.3.5 change-type classifier using only (ΔFR(s), sampling_ratio), omitting the failure-prediction model and behavioral impact scores. If either baseline reaches CASPER's Table 15 AUC within, say, 0.01, the behavioral-signal component is not load-bearing and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ3 claim is that CASPER's behavioral-signal pipeline prioritizes regressed slices better than random. But in §2.3.5 the change-type classifier is trained on two features: ΔFR(s), the failure-rate change measured on K executed representatives, and sampling_ratio. In §3.4.3 the ground-truth label for 'regressed' is the sign of the same failure-rate change computed over the entire slice. Thus the primary feature is a noisy version of the label itself. Because CASPER-SI enforces output consistency ≈97% (Table 7), even K=1–4 representatives are highly predictive of the full-slice failure-rate change. That is enough to explain the high held-out ROC-AUC in Table 14 and the Recall@K gains in Table 15. The only baseline is random ranking; there is no ablation that removes behavioral signals, no baseline that ranks directly by sampled ΔFR(s), and no comparison against a classifier that uses only ΔFR(s)+sampling_ratio without the failure-prediction model. The reported results therefore do not establish that behavioral signals or the failure-prediction model add value; they may simply confirm that sampling a nearly consistent slice's outcomes estimates its own label. This is a construct-validity threat to the strongest claim, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CASPER, a two-stage framework for regression testing of LLM-based systems. Stage 1 (CASPER-SI) partitions a regression test suite into slices using an NSGA-II multi-objective search over behavioral signals extracted from agent conversation logs, optimizing output consistency, intra-slice cohesion, and inter-slice separation. Stage 2 ranks the slices after a system change by (a) learning a failure prediction model from V0 behavioral signals, (b) executing K representative instances per slice on V1, (c) computing a change in failure rate, ΔFR(s), and a sampling ratio, (d) training regression/improvement classifiers on these two features, and (e) ordering slices by predicted category (regressed, improved, unchanged) and then by aggregated change in failure impact score. The evaluation uses SWE-bench Verified with four change scenarios, comparing slice identification against GMM and HDBSCAN and ranking against a random baseline. The paper reports high output consistency for CASPER-SI and consistent Recall@K/AUC improvements over random ranking.","tokens_in":35888,"tokens_out":7825,"duration_ms":76204,"significance":"If the prioritization claim were valid, the paper would address a timely and practical problem: focusing limited regression-testing effort on the slices most likely to regress in LLM-based systems. The slice-identification contribution is reasonably supported: the OC gains over clustering baselines are large, statistically tested, and accompanied by a substantial data-collection effort. The framing of regression granularity for LLM-based systems is also well motivated. However, the central Stage 2 claim is undermined by a construct-validity problem: the primary classifier feature is a sampled estimate of the ground-truth label, and the only baseline is random ranking. As a result, the reported prioritization gains do not establish the role of behavioral signals or the failure prediction model, and the main contribution of the paper is not supported by the evidence as presented.","major_comments":[{"comment":"The change-type classifier in §2.3.5 is trained on two features, ΔFR(s) and sampling_ratio, where ΔFR(s) is the failure-rate change computed on the K executed representatives per slice. The ground-truth label in §3.4.3(d) is defined by the sign of the same failure-ratio change computed over the entire slice. Because CASPER-SI produces slices with mean output consistency above 97% (Table 7), the K-sample failure-rate change is very likely to agree in sign with the full-slice label; the feature is thus a near-deterministic estimate of the label. This explains the held-out ROC-AUC values in Table 14 and the Recall@K improvements in Table 15 without invoking behavioral signals or the failure prediction model. The manuscript must include a baseline that ranks slices directly by the sampled ΔFR(s) and an ablation that removes the failure prediction model/behavioral-signal component; without these, the RQ3 claim is not established.","section":"§2.3.5, §3.4.3"},{"comment":"Behavioral signals are not features of the change-type classifier; the classifier uses only ΔFR(s) and sampling_ratio. Since §2.3.7 prioritizes all predicted regressed (R) slices ahead of improved (I) and unchanged (NC) slices, the Recall@K behavior is driven by this category classifier. Behavioral signals enter only when ordering slices within the same category via the aggregated failure-impact score (§2.3.6). As a result, the paper's characterization of the ranking as based on behavioral information extracted from execution logs (abstract, §1.2) is not supported by the architecture as described. The authors should either incorporate behavioral features into the change-type classifier or evaluate and report within-category ranking quality separately.","section":"§2.3.5, §2.3.6, §2.3.7"},{"comment":"The RQ3 evaluation compares only against random ranking and reports no confidence intervals or significance tests for the AUC differences. Given the leakage described above, random is a weak baseline: it does not isolate the contribution of ΔFR(s), sampling_ratio, the failure prediction model, or the behavioral signals. The paper also does not report a comparison with the trivial strategy of ranking slices by their sampled ΔFR(s). Adding such baselines and accompanying statistical tests is necessary before the claim that CASPER 'improves regressed slice prioritization' can be accepted.","section":"RQ3, Table 15, Fig. 2"}],"minor_comments":[{"comment":"The heading 'Slice ranking (CASPER-SI)' reuses the CASPER-SI acronym from the slice-identification stage; a distinct name such as CASPER-SR or CASPER-Rank would avoid confusion.","section":"§2.3 heading"},{"comment":"Equation (16) defines the individual-level inter-slice objective using the symbol InterCD, whereas the surrounding text and Equation (15) use Inter_SD; this is likely a typo.","section":"Equation (16)"},{"comment":"The grouping 'Model change & prompt change one' should clarify whether these are the same slice partition evaluated under two change conditions or two separately generated partitions; the current labeling is ambiguous.","section":"Tables 6 and 7"},{"comment":"The ACM Reference Format block lists the date as August 2018 and the current year as 2026; this appears to be a template artifact that should be corrected.","section":"ACM Reference Format"},{"comment":"The construct-validity discussion acknowledges that behavioral signals are a proxy, but it does not mention the more direct threat that ΔFR(s) is a sampled estimate of the ground-truth label; this circularity should be addressed explicitly.","section":"Section 5 (threats to validity)"},{"comment":"The replication-package statement says the package will be provided 'after accepetance for publication'; besides the typo, providing the data and code at submission time would substantially strengthen verifiability.","section":"Replication package"}],"recommendation":"reject","confidential_remarks":"The slice-identification part is reasonable and could be a publishable contribution on its own. The prioritization evaluation, however, is invalid as designed: the main classifier feature is a sampled estimate of the label, and no ablation or feature-only baseline is provided. This is more than a presentation issue, because the paper's central claim about behavioral-signal-based ranking is not supported. I would be willing to reconsider a major revision that adds a ΔFR-only baseline, removes the failure-prediction model in an ablation, and either integrates behavioral signals into the change-type classifier or substantially narrows the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, CASPER-SI, the slice identification stage, is a genuine contribution: multiobjective evolutionary slicing over behavioral signals beat GMM and HDBSCAN on output consistency (97%+ vs ~80%) with statistical tests, while keeping coherence competitive. That part is credible. Second, the slice ranking evaluation has a load-bearing circularity. In §2.3.5 the change-type classifier uses ΔFR(s) — the failure-rate change measured on K executed representatives — as its primary feature. In §3.4.3 the ground-truth 'regressed' label is defined as full-slice failure-ratio change > 0. So the feature is a sampled estimate of the label. Because CASPER-SI enforces ~97% output consistency, even K=1–4 samples estimate the full-slice failure rate well. The RQ3 gains over random ranking are then almost certainly from ΔFR(s) reading its own label, not from behavioral signals. There is no baseline that ranks by raw ΔFR(s) alone, and no ablation that removes behavioral signals. The failure prediction model only orders slices within the same category, so it is not the driver.\n\nCredit where due: the paper is honest about ΔFR being the 'primary signal' in §2.3.5, but it doesn't confront the circularity with the ground truth. The behavioral signals are thoughtfully engineered, and the end-to-end framing is new relative to prior slice discovery and LLM regression testing work. The LODO evaluation for classifier selection is good practice. The prompt-change datasets are a nice touch.\n\nRecommendation: I would not accept in current form. The slice identification half could be published after repositioning. The ranking half needs a ΔFR-only baseline and an ablation showing behavioral signals add value. Replication artifacts are promised only after acceptance, which is another blocker. Still, this deserves a serious referee — not a desk reject. A good reviewer could reshape it into a useful paper.","headline":"Solid slice identification, but the ranking evaluation's main feature is a sampled estimate of its own label—so the headline result isn't established.","tokens_in":36280,"tokens_out":3925,"would_cite":false,"duration_ms":34203,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CASPER ranks test slices by regression likelihood, so a short run of representative instances finds the behaviors a model or prompt change broke.","keywords":["LLM regression testing","slice prioritization","test suite slicing","behavioral signal extraction","evolutionary multi-objective search","agent execution logs","regression slice ranking","software issue resolution"],"falsifier":"For a held-out change, compute each slice's failure-rate change using every instance and compare that full-slice value with the value obtained from only the K sampled instances; if the rank correlation is near zero for K=1 or K=2 while CASPER still reports high Recall@K, the reported ranking benefit cannot be coming from the sampled estimate.","tokens_in":35289,"feed_emoji":"🧪","tokens_out":7789,"duration_ms":68380,"temperature":0.7,"pith_summary":"Regression testing of LLM-based systems has a granularity problem: a single failing prompt is not reliable evidence of a regression, and an overall performance drop does not say which behaviors broke. The right unit, this paper argues, is the slice, a group of semantically related regression instances on which the system performs consistently. CASPER builds slices with an evolutionary search and, after a prompt or model change, executes only one to four representative instances per slice, reads behavioral signals from agent logs, and ranks slices by predicted regression likelihood. The paper's claim is that this ranking is consistently more effective than random ordering across model and prompt changes, with the largest gains under small testing budgets, so most regressed slices can be found by testing a small fraction of the suite.","feed_headline":"Slice ranking beats random for LLM regression testing","feed_subtitle":"CASPER executes a handful of representative cases per slice and still surfaces regressed slices early.","key_machinery":"The load-bearing object is the slice: a partition of regression instances that are semantically coherent in a behavioral-signal space and over which the LLM-based system is predominantly successful or predominantly unsuccessful. Slice identification is cast as a multi-objective optimization solved by an evolutionary genetic search whose three objectives are output inconsistency, intra-slice distance, and inter-slice distance, with custom crossover and mutation operators that respect minimum and maximum slice sizes. The ranking stage uses ten behavioral signals extracted from agent conversation logs, including file-level recall, mean absolute state movement, trajectory entropy deviation, testing effort, test failure rate, dominant file ratio, cross-state file overlap, and three plan-compliance metrics. A failure prediction model trained on the original version assigns each instance a failure impact score; after executing representative instances on the changed version, the change in that score, aggregated per slice and gated by a regression/improvement classifier, produces the final ranked order.","core_discovery":"The central discovery, stated the way the paper argues it, is that the behavioral footprint of a change, visible in the agent's conversation logs, is sufficient to rank whole slices by regression risk without exhaustive re-execution. On the original version, CASPER learns a failure prediction model from ten behavioral signals and partitions the test suite into slices by jointly minimizing output inconsistency and intra-slice distance while maximizing inter-slice distance. After a change, it executes K representative instances per slice, extracts the same signals, and computes the change in the model's predicted failure impact relative to the original version. Slices are classified as regressed, improved, or unchanged, then sorted so regressed slices come first. Across four change datasets and K from one to four, the Recall@K curves consistently lie between random ranking and the ideal curve, with AUC improvements over random ranking between roughly 8 and 49 percent.","pith_inferences":["Left implicit is a control the authors did not run: perturb the behavioral signals randomly and show that Recall@K degrades; that experiment would separate the signal's contribution from any label leakage in the classifier.","The same slice-ranking machinery could be applied to any agentic system with logged tool trajectories, such as browser automation or document QA, but the ten signals would need to be re-derived for those domains; the paper's framework is the process, not a fixed feature set.","If the small-sample failure-rate estimate is noisy, the two-feature classifier may partly be learning to trust slices with larger sampling ratios; a testable extension is to replace the raw failure-rate change with a sample-size-normalized version and measure whether ranking holds."],"forward_implications":["Executing only one representative instance per slice is enough to lift ranking well above random on every dataset the paper evaluates, so the largest cost saving is available at the tightest budget.","Prompt changes are the easiest regressions to find under this approach; AUC gains reach roughly 49 percent, so CI/CD pipelines that touch prompts get the most value from slice ranking.","Same-family model changes produce subtler behavioral shifts and smaller gains, implying that teams upgrading within one model family should expect to sample more instances per slice.","Because the framework ranks slices before any oracle runs on the whole suite, it can be used as a triage step: run top-ranked slices, inspect their failures, and only expand execution if a regression is confirmed."],"supporting_citations":[{"why":"Motivates slice-level regression testing for LLM-based systems and defines slices as the appropriate granularity between single instances and the whole suite.","marker":"[17]"},{"why":"Supplies the slice quality requirements of semantic coherence and consistent performance that CASPER's optimization objectives formalize.","marker":"[9]"},{"why":"Provides the plan-compliance metrics adopted as behavioral signals and the empirical connection between plan adherence and patch success.","marker":"[15]"},{"why":"Supplies the 500-issue regression dataset used to evaluate slice identification and ranking.","marker":"[12]"},{"why":"Provides the open-source issue-resolution agent whose conversation logs and evaluation harness ground the case study.","marker":"[21]"},{"why":"Supplies the reasoning-and-acting interaction pattern that justifies extracting per-turn thoughts, actions, and tool results from logs.","marker":"[22]"},{"why":"Provides the multi-objective evolutionary algorithm behind the slice identification search.","marker":"[6]"},{"why":"Provides the half-uniform crossover operator adapted for the slice-assignment genome in the genetic search.","marker":"[8]"},{"why":"One of the clustering baselines, Gaussian mixture models, used for the slice identification comparison.","marker":"[3]"},{"why":"The other clustering baseline, hierarchical density-based clustering, used for the slice identification comparison.","marker":"[18]"}],"fun_headline_variants":["Rank slices, not runs: CASPER reorders LLM regression tests","LLM regression testing: Prioritize slices, not individual failures","CASPER targets likely regressed slices with few test executions","Slice-level ranking boosts LLM regression discovery up to 49%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking collapses if the failure-rate change measured on one to four representative instances per slice is not a faithful stand-in for how the entire slice behaves after the change.","fun_headline_variants_meta":{"raw":{"variants":["Rank slices, not runs: CASPER reorders LLM regression tests","LLM regression testing: Prioritize slices, not individual failures","CASPER targets likely regressed slices with few test executions","Slice-level ranking boosts LLM regression discovery up to 49%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3216,"prompt_tokens":928,"completion_tokens":2288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2215}},"tokens_in":544,"tokens_out":2288,"duration_ms":14641,"temperature":1.0,"reasoning_tokens":2215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:21:03.729203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a held-out change, compute each slice's failure-rate change using every instance and compare that full-slice value with the value obtained from only the K sampled instances; if the rank correlation is near zero for K=1 or K=2 while CASPER still reports high Recall@K, the reported ranking benefit cannot be coming from the sampled estimate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates slice-level regression testing for LLM-based systems and defines slices as the appropriate granularity between single instances and the whole suite."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-objective evolutionary algorithm behind the slice identification search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the half-uniform crossover operator adapted for the slice-assignment genome in the genetic search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The other clustering baseline, hierarchical density-based clustering, used for the slice identification comparison."}],"review_version":2}