Pith. sign in

REVIEW 3 major objections 6 minor 27 references

CASPER-Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read CASPER ranks test slices by regression likelihood, so a short run of representative instances finds the behaviors a model or prompt change broke.

desk verdict Solid slice identification, but the ranking evaluation's main feature is a sampled estimate of its own label—so the headline result isn't established. read the letter →

arxiv 2608.00378 v1 pith:7VF52G5B submitted 2026-08-01 cs.SE

classification cs.SE
keywords LLMregressiontestingsliceprioritizationtestsuiteslicingbehavioralsignalextractionevolutionarymulti-objectivesearchagentexecutionlogsrankingsoftwareissueresolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Regression testing of LLM-based systems has a granularity problem: a single failing prompt is not reliable evidence of a regression, and an overall performance drop does not say which behaviors broke. The right unit, this paper argues, is the slice, a group of semantically related regression instances on which the system performs consistently. CASPER builds slices with an evolutionary search and, after a prompt or model change, executes only one to four representative instances per slice, reads behavioral signals from agent logs, and ranks slices by predicted regression likelihood. The paper's claim is that this ranking is consistently more effective than random ordering across model and prompt changes, with the largest gains under small testing budgets, so most regressed slices can be found by testing a small fraction of the suite.

What carries the argument

The load-bearing object is the slice: a partition of regression instances that are semantically coherent in a behavioral-signal space and over which the LLM-based system is predominantly successful or predominantly unsuccessful. Slice identification is cast as a multi-objective optimization solved by an evolutionary genetic search whose three objectives are output inconsistency, intra-slice distance, and inter-slice distance, with custom crossover and mutation operators that respect minimum and maximum slice sizes. The ranking stage uses ten behavioral signals extracted from agent conversation logs, including file-level recall, mean absolute state movement, trajectory entropy deviation, testing effort, test failure rate, dominant file ratio, cross-state file overlap, and three plan-compliance metrics. A failure prediction model trained on the original version assigns each instance a failure impact score; after executing representative instances on the changed version, the change in that score, aggregated per slice and gated by a regression/improvement classifier, produces the final ranked order.

What would settle it

For a held-out change, compute each slice's failure-rate change using every instance and compare that full-slice value with the value obtained from only the K sampled instances; if the rank correlation is near zero for K=1 or K=2 while CASPER still reports high Recall@K, the reported ranking benefit cannot be coming from the sampled estimate.

Watch

Extended reading notes

Core claim

The central discovery, stated the way the paper argues it, is that the behavioral footprint of a change, visible in the agent's conversation logs, is sufficient to rank whole slices by regression risk without exhaustive re-execution. On the original version, CASPER learns a failure prediction model from ten behavioral signals and partitions the test suite into slices by jointly minimizing output inconsistency and intra-slice distance while maximizing inter-slice distance. After a change, it executes K representative instances per slice, extracts the same signals, and computes the change in the model's predicted failure impact relative to the original version. Slices are classified as regressed, improved, or unchanged, then sorted so regressed slices come first. Across four change datasets and K from one to four, the Recall@K curves consistently lie between random ranking and the ideal curve, with AUC improvements over random ranking between roughly 8 and 49 percent.

Load-bearing premise

The whole ranking collapses if the failure-rate change measured on one to four representative instances per slice is not a faithful stand-in for how the entire slice behaves after the change.

Editorial extensions

If this is right

  • Executing only one representative instance per slice is enough to lift ranking well above random on every dataset the paper evaluates, so the largest cost saving is available at the tightest budget.
  • Prompt changes are the easiest regressions to find under this approach; AUC gains reach roughly 49 percent, so CI/CD pipelines that touch prompts get the most value from slice ranking.
  • Same-family model changes produce subtler behavioral shifts and smaller gains, implying that teams upgrading within one model family should expect to sample more instances per slice.
  • Because the framework ranks slices before any oracle runs on the whole suite, it can be used as a triage step: run top-ranked slices, inspect their failures, and only expand execution if a regression is confirmed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Left implicit is a control the authors did not run: perturb the behavioral signals randomly and show that Recall@K degrades; that experiment would separate the signal's contribution from any label leakage in the classifier.
  • The same slice-ranking machinery could be applied to any agentic system with logged tool trajectories, such as browser automation or document QA, but the ten signals would need to be re-derived for those domains; the paper's framework is the process, not a fixed feature set.
  • If the small-sample failure-rate estimate is noisy, the two-feature classifier may partly be learning to trust slices with larger sampling ratios; a testable extension is to replace the raw failure-rate change with a sample-size-normalized version and measure whether ranking holds.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CASPER, a two-stage framework for regression testing of LLM-based systems. Stage 1 (CASPER-SI) partitions a regression test suite into slices using an NSGA-II multi-objective search over behavioral signals extracted from agent conversation logs, optimizing output consistency, intra-slice cohesion, and inter-slice separation. Stage 2 ranks the slices after a system change by (a) learning a failure prediction model from V0 behavioral signals, (b) executing K representative instances per slice on V1, (c) computing a change in failure rate, ΔFR(s), and a sampling ratio, (d) training regression/improvement classifiers on these two features, and (e) ordering slices by predicted category (regressed, improved, unchanged) and then by aggregated change in failure impact score. The evaluation uses SWE-bench Verified with four change scenarios, comparing slice identification against GMM and HDBSCAN and ranking against a random baseline. The paper reports high output consistency for CASPER-SI and consistent Recall@K/AUC improvements over random ranking.

Significance. If the prioritization claim were valid, the paper would address a timely and practical problem: focusing limited regression-testing effort on the slices most likely to regress in LLM-based systems. The slice-identification contribution is reasonably supported: the OC gains over clustering baselines are large, statistically tested, and accompanied by a substantial data-collection effort. The framing of regression granularity for LLM-based systems is also well motivated. However, the central Stage 2 claim is undermined by a construct-validity problem: the primary classifier feature is a sampled estimate of the ground-truth label, and the only baseline is random ranking. As a result, the reported prioritization gains do not establish the role of behavioral signals or the failure prediction model, and the main contribution of the paper is not supported by the evidence as presented.

major comments (3)
  1. [§2.3.5, §3.4.3] The change-type classifier in §2.3.5 is trained on two features, ΔFR(s) and sampling_ratio, where ΔFR(s) is the failure-rate change computed on the K executed representatives per slice. The ground-truth label in §3.4.3(d) is defined by the sign of the same failure-ratio change computed over the entire slice. Because CASPER-SI produces slices with mean output consistency above 97% (Table 7), the K-sample failure-rate change is very likely to agree in sign with the full-slice label; the feature is thus a near-deterministic estimate of the label. This explains the held-out ROC-AUC values in Table 14 and the Recall@K improvements in Table 15 without invoking behavioral signals or the failure prediction model. The manuscript must include a baseline that ranks slices directly by the sampled ΔFR(s) and an ablation that removes the failure prediction model/behavioral-signal component; without these, the RQ3 claim is not established.
  2. [§2.3.5, §2.3.6, §2.3.7] Behavioral signals are not features of the change-type classifier; the classifier uses only ΔFR(s) and sampling_ratio. Since §2.3.7 prioritizes all predicted regressed (R) slices ahead of improved (I) and unchanged (NC) slices, the Recall@K behavior is driven by this category classifier. Behavioral signals enter only when ordering slices within the same category via the aggregated failure-impact score (§2.3.6). As a result, the paper's characterization of the ranking as based on behavioral information extracted from execution logs (abstract, §1.2) is not supported by the architecture as described. The authors should either incorporate behavioral features into the change-type classifier or evaluate and report within-category ranking quality separately.
  3. [RQ3, Table 15, Fig. 2] The RQ3 evaluation compares only against random ranking and reports no confidence intervals or significance tests for the AUC differences. Given the leakage described above, random is a weak baseline: it does not isolate the contribution of ΔFR(s), sampling_ratio, the failure prediction model, or the behavioral signals. The paper also does not report a comparison with the trivial strategy of ranking slices by their sampled ΔFR(s). Adding such baselines and accompanying statistical tests is necessary before the claim that CASPER 'improves regressed slice prioritization' can be accepted.
minor comments (6)
  1. [§2.3 heading] The heading 'Slice ranking (CASPER-SI)' reuses the CASPER-SI acronym from the slice-identification stage; a distinct name such as CASPER-SR or CASPER-Rank would avoid confusion.
  2. [Equation (16)] Equation (16) defines the individual-level inter-slice objective using the symbol InterCD, whereas the surrounding text and Equation (15) use Inter_SD; this is likely a typo.
  3. [Tables 6 and 7] The grouping 'Model change & prompt change one' should clarify whether these are the same slice partition evaluated under two change conditions or two separately generated partitions; the current labeling is ambiguous.
  4. [ACM Reference Format] The ACM Reference Format block lists the date as August 2018 and the current year as 2026; this appears to be a template artifact that should be corrected.
  5. [Section 5 (threats to validity)] The construct-validity discussion acknowledges that behavioral signals are a proxy, but it does not mention the more direct threat that ΔFR(s) is a sampled estimate of the ground-truth label; this circularity should be addressed explicitly.
  6. [Replication package] The replication-package statement says the package will be provided 'after accepetance for publication'; besides the typo, providing the data and code at submission time would substantially strengthen verifiability.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ3 gains are largely explained by the ΔFR(s) classifier feature being a sampled estimate of the ground-truth regression label; behavioral-signal contribution is not separately demonstrated.

  1. fitted input called prediction [Sections 2.3.5, 3.4.3 (with Table 7)]
    "The first feature used by the classification model is the change in failure rate, ΔFR(s), which captures the observed behavioral difference between the two versions for slice s: ΔFR(s)=FR_V1(s)−FR_V0(s) ... We select ΔFR(s) as a feature because it directly represents the observed impact signal used to identify changes in slice behavior. ... We label a slice as 'regressed' if its change in failure ratio is greater than 0."

    The ground-truth regression label is defined as the sign of the full-slice change in failure ratio (3.4.3), while the classifier's primary feature is the same failure-rate change computed on the K executed representatives (2.3.5). Both quantities are aggregates of the same binary patch_success outcomes, differing only in denominator (K vs. slice size). Because CASPER-SI enforces ~97% output consistency (Table 7), K=1-4 sampled instances are near-sufficient to determine the full-slice failure-rate direction, so the feature is a noisy estimate of the label itself. The only baseline is random ranking; there is no ablation removing ΔFR, no baseline that ranks by sampled ΔFR alone, and no classifier variant using behavioral signals without ΔFR.

full rationale

CASPER is largely self-contained: slice identification is evaluated against GMM/HDBSCAN on SWE-bench data, and the behavioral-signal failure-prediction model is trained on V0 logs, which is not circular. The self-citation pattern is absent (the cited plan-compliance work [15] has disjoint authors and is used as an input feature, not as a proof of the central claim). The central circularity is confined to Stage 2's change-type classification: the regressed/not-regressed label (3.4.3) is the sign of the full-slice failure-rate change, and the classifier's primary feature ΔFR(s) (2.3.5) is the same failure-rate change measured on K representatives. Given the high output consistency of CASPER-SI slices (mean OC >97%, Table 7), the feature is an almost direct estimate of the label, making high classification ROC-AUC and Recall@K gains partly tautological. Because no ablation isolates the behavioral-signal contribution or compares against a ΔFR-only ranking baseline, the paper's strongest claim that behavioral signals drive prioritization is not established independently. This is partial circularity rather than full equivalence, since within-category ranking still uses the behavioral failure-impact score and the slice-identification stage has independent content.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain-specific behavioral signal design, tuned hyperparameters, and the assumption that sampled failure-rate changes reliably represent slice-level impact. No new physical or conceptual entities are introduced; slices are groupings of existing test instances and behavioral signals are derived features. The ranking classifier's use of Delta FR as a feature while the label is defined by failure-ratio change creates a circularity burden rather than an axiom.

free parameters (8)
  • Population_size = 136 (model change and prompt change one), 93 (same-family), 96 (prompt change two)
    Tuned per dataset with Optuna; affects NSGA-II search diversity and slice identification outcomes.
  • Crossover_prob = 0.994, 0.894, 0.823 across datasets
    Hyperparameter tuned per dataset; controls genetic diversity in slice search.
  • Mutation_prob = 0.219, 0.375, 0.468 across datasets
    Tuned per dataset; controls exploration of slice partitions.
  • Max_generations = 6907, 5710, 9423 across datasets
    Tuned per dataset; sets the NSGA-II computational budget.
  • Max_mutations = 2, 1, 1 across datasets
    Tuned per dataset; bounds move and swap operations per mutation.
  • Min_slice_size = 5 (all datasets)
    Constraint on minimum slice size; chosen from search range [5,10].
  • Max_slice_size = 19, 16, 17 across datasets
    Constraint on maximum slice size; chosen from search range [10,20].
  • Failure prediction and impact classifier choices = LR (C=1.0, max_iter=1000), SVM (RBF), Random Forest with defaults
    Selected based on preliminary experiments and leave-one-dataset-out model selection; these choices affect the final rankings.
assumptions (5)
  • domain assumption Behavioral signals extracted from agent conversation logs are predictive of patch success and remain predictive after model or prompt changes.
    Introduced in Section 2.2.1; the entire ranking pipeline depends on these features capturing task-relevant behavior.
  • domain assumption Representative instances sampled per slice generalize to the rest of the slice.
    Used in Sections 2.3.2 and 2.3.5; with K between 1 and 4, the sampled failure-rate change is assumed to represent the full slice.
  • domain assumption Slice-based regression granularity, defined by output consistency and semantic coherence, is the correct unit for LLM regression testing.
    Motivated by Ma et al. [17] in Section 1.2; the entire framework is built around this premise.
  • domain assumption Gold patch information is available for computing file-level recall and for evaluating generated patches.
    Section 2.2.1 computes file-level recall using the gold patch; this holds for SWE-bench but may not hold in real deployments without ground-truth patches.
  • domain assumption NSGA-II, with the specified encoding, crossover, mutation, and repair operators, converges to useful slice partitions.
    Section 2.2.3 relies on the evolutionary search without formal convergence guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CASPER-Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems." pith.science (2026). https://pith.science/paper/7VF52G5B

@misc{pith2026260800378,
  author       = {Pith},
  title        = {Pith review of: CASPER-Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VF52G5B}},
  note         = {Machine review of arXiv:2608.00378}
}
read the original abstract

Regression testing for LLM-based systems poses unique challenges because individual regression instances provide limited information about system-level regressions. A failure in a single instance does not necessarily indicate a meaningful regression or provide sufficient information to diagnose affected behaviors. Conversely, detecting regressions based only on overall system performance changes is too coarse-grained, as it does not identify which behaviors are affected. This motivates analyzing regression instances at an intermediate level through test suite slices. To address this challenge, we propose CASPER, a change-aware slice prioritization framework for efficient regression testing of prompt-level and model-level changes in LLM-based systems. CASPER first identifies regression slices containing semantically related instances with consistent performance characteristics using an evolutionary slice identification approach. Given a change to an LLM-based application, CASPER prioritizes slices according to their likelihood of regression using behavioral information extracted from execution logs. We instantiate CASPER in the software issue resolution domain and evaluate slice identification against clustering-based baselines and regressed slice prioritization against a random ranking baseline. Results show that CASPER generates more consistent slices while maintaining comparable or better semantic coherence and improves regressed slice prioritization across different LLM-based system changes and testing budgets.

Figures

Figures reproduced from arXiv: 2608.00378 by the authors.

Figure 1
Figure 1. Overview of CASPER. A regression testing dataset is first executed on the LLMUT (V0) to collect execution result and conversation logs, from which behavioral signals are extracted. Stage 1 identifies slices, while Stage 2 ranks the slices according to their likelihood of being impacted after a system change, producing a prioritized list of slices. 2.2 Stage 1: Slice identification (CASPER-SI) Stage 1, the slice iden… view at source ↗
Figure 2
Figure 2. Recall@K curves for CASPER and the baseline under different numbers of samples per slice. The x-axis is the top-𝑘 percentage obtained by dividing 𝐾 by the total number of slices in the ground-truth dataset. n denotes the number of samples per slice. shifts produce more pronounced changes in the slice-level behavioral features used by CASPER, make impacted slices easier to distinguish and prioritize [PITH_FULL_IMAGE… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages

  1. [1]

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

  2. [2]

    Khalil Amine. 2019. Multiobjective simulated annealing: Principles and algorithm variants.Advances in Operations Research2019, 1 (2019), 8134674

  3. [3]

    2006.Pattern recognition and machine learning

    Christopher M Bishop and Nasser M Nasrabadi. 2006.Pattern recognition and machine learning. Vol. 4. Springer

  4. [4]

    Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. 2019. Slice finder: Automated data slicing for model validation. In2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 1550–1553

  5. [5]

    Kalyanmoy Deb. 2011. Multi-objective optimisation using evolutionary algorithms: an introduction. InMulti-objective evolutionary optimisation for product design and manufacturing. Springer, 3–34

  6. [6]

    Kalyanmoy Deb and Himanshu Jain. 2013. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints.IEEE transactions on evolutionary computation18, 4 (2013), 577–601

  7. [7]

    Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A systematic review on regression test selection techniques.Information and Software Technology52, 1 (2010), 14–30

  8. [8]

    Larry J Eshelman. 1991. The CHC adaptive search algorithm: How to have safe search when engaging in nontraditional genetic recombination. In Foundations of genetic algorithms. Vol. 1. Elsevier, 265–283

Show all 27 references
  1. [9]

    Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. 2022. Domino: Discovering systematic errors with cross-modal embeddings.arXiv preprint arXiv:2203.14960(2022)

  2. [10]

    Shantanu Ghosh, Rayan Syed, Chenyu Wang, Vaibhav Choudhary, Binxu Li, Clare B Poynton, Shyam Visweswaran, and Kayhan Batmanghelich

  3. [11]

    Wenyue Hua, Lifeng Jin, Linfeng Song, Haitao Mi, Yongfeng Zhang, and Dong Yu. 2022. Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing.arXiv preprint arXiv:2211.04476(2022)

  4. [12]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?...

  5. [13]

    Nari Johnson, Ángel Alexander Cabrera, Gregory Plumb, and Ameet Talwalkar. 2023. Where does my model underperform? a human evaluation of slice discovery algorithms. InProceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 11. 65–76

  6. [14]

    2020.Mining of massive data sets

    Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. 2020.Mining of massive data sets. Cambridge university press. Manuscript submitted to ACM CASPER — Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems 41

  7. [15]

    Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand. 2026. Evaluating Plan Compliance in Autonomous Programming Agents.arXiv preprint arXiv:2604.12147(2026)

  8. [16]

    2013.Essentials of Metaheuristics(second ed.)

    Sean Luke. 2013.Essentials of Metaheuristics(second ed.). Lulu. Available for free at http://cs.gmu.edu/∼sean/book/metaheuristics/

  9. [17]

    Wanqin Ma, Chenyang Yang, and Christian Kästner. 2024. (why) is my prompt getting worse? Rethinking regression testing for evolving llm apis. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 166–171

  10. [18]

    Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering.Journal of Open Source Software2, 11 (2017),

  11. [19]

    Vincent Olesen, Nina Weng, Aasa Feragen, and Eike Petersen. 2024. Slicing through bias: explaining performance gaps in medical image analysis using slice discovery methods. InMICCAI Workshop on Fairness of AI in Medical Imaging. Springer, 3–13

  12. [20]

    Gregory Plumb, Nari Johnson, Ángel Alexander Cabrera, and Ameet Talwalkar. 2022. Towards a more rigorous science of blindspot discovery in image classification models.arXiv preprint arXiv:2207.04104(2022)

  13. [21]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. ht...

  14. [22]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629(2022)

  15. [23]

    Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey.Software testing, verification and reliability 22, 2 (2012), 67–120

  16. [24]

    Xiaoyu Zhang, Jorge Piazentin Ono, Huan Song, Liang Gou, Kwan-Liu Ma, and Liu Ren. 2022. SliceTeller: A data slice-driven approach for machine learning model validation.IEEE Transactions on Visualization and Computer Graphics29, 1 (2022), 842–852

  17. [25]

    command":

    Eckart Zitzler and Simon Künzli. 2004. Indicator-based selection in multiobjective search. InInternational conference on parallel problem solving from nature. Springer, 832–842. A Appendix A.1 Prompt Diff file for the prompt change dataset one diff --git a/src/minisweagent/con...

  18. [205]

    doi:10.21105/joss.00205

  19. [2025]

    InFindings of the Association for Computational Linguistics: ACL 2025

    LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers. InFindings of the Association for Computational Linguistics: ACL 2025. 22935–22970

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.