{"id":"6827a9ae-ea6c-4ef2-bafa-3c60db294d0c","arxiv_id":"2501.11550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement-learning test prioritizer with pipeline-specific rewards finds the first failing test within 16% of the suite on average and selects 87% of relevant post-submit transitions with a 50% time budget.","lead":"This paper describes an AI system that decides which software tests to run first and which to skip in large continuous integration pipelines. On BMW's 70-million-line codebase, it claims to reveal failures within the first 16% of the test suite and to catch 87% of important post-submit transitions using half the test time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-submit 87%/99.78% claims are measured against a flaky-transition label (Section IV.C: three-cycle window) that is also built into the CostChangeRank reward (Eq.","rationale":"The reader’s weakest_assumption identifies exactly the same construct-validity threat: the flaky-transition definition is simultaneously the training reward and the evaluation ground truth. My stress test found no independent flaw that is more decisive. The paper has real strengths: the 20-week industrial dataset is released, the replay methodology avoids disrupting production, and Section VI honestly discusses safeguards needed before deployment. However, the post-submit headline numbers are conditional on a threshold that is tuned to the same data, and the released dataset makes a sensitivity test feasible. Since the post-submit claim is meaningfully conditional and the pre-submit claim is largely unaffected, the appropriate verdict remains CONDITIONAL; no adjustment to the reader’s verdict is needed.","tokens_in":15653,"tokens_out":10396,"duration_ms":121465,"concrete_test":"Re-run the post-submit replay used for Figures 7 and 8 with the flaky-transition window θ varied over {1, 2, 5, 10, 20} CI cycles, keeping the rest of the training setup fixed, and report CostChangeRank, RNChange, and Random at the 50% budget: per-cycle detection ratio and cumulative detection by five cycles. If the 87% or 99.78% figures, or the ranking of CostChangeRank versus baselines, moves by more than about 5 percentage points across θ, the headline numbers are artifacts of the θ=3 choice; if the curves are flat across θ, the concern is resolved and the conditional verdict can be strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central post-submit claim in RQ2 equates “developer-relevant” with “non-flaky transition,” and flakiness is operationalized as switching back within three CI cycles (Section IV.C). This same three-cycle rule is used twice: as the ground-truth label for Figures 7 and 8, and as the reward label in CostChangeRank, where flaky transitions are penalized with -1 and relevant transitions rewarded with +1 (Eq. 2). The threshold is chosen from the churn rate of the same dataset, with Figure 6 showing that many transitions revert quickly, but nothing validates that three cycles separates true regressions/bug fixes from flaky tests. If a flaky test stabilizes only after 4+ cycles, it will be mislabeled as relevant; if a real regression is fixed within three cycles, it will be mislabeled as flaky. Since the reported 87% at a 50% budget and 99.78% within five cycles are computed against this self-generated label, they are conditional on the threshold and may not reflect developer-confirmed relevance. The pre-submit RQ1 results are less affected because NAPFD, NFR, and NTTF are standard failure-ranking metrics, although they apply only to failed cycles.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PR-DQL, a Deep-Q-Learning-based test prioritization and selection approach for large-scale CI systems, with two pipeline-specific reward functions: CostRank for pre-submit pipelines and CostChangeRank for post-submit pipelines. Using only language-agnostic features (historical test results, test-name embeddings, recency, and average duration), the authors evaluate on a 20-week industrial dataset from BMW, replaying CI cycles against RANDOM, ROCKET, COLEMAN, and a custom RNChange reward baseline. The central claims are that PR-DQL schedules the first failing test within the first 16% of tests in pre-submit pipelines, and that in post-submit pipelines it selects 87% of developer-relevant transitions within half the execution time and detects 99.78% of relevant transitions within five CI cycles. The paper contributes an open dataset and a lightweight, language-agnostic approach, but the post-submit evaluation relies on a self-defined flaky-transition threshold that is used both as the training reward label and as the evaluation ground truth, and the comparative claims lack significance testing.","tokens_in":15827,"tokens_out":4652,"duration_ms":50353,"significance":"If the results hold, this is a valuable industrial contribution: the feature set is genuinely lightweight and language-agnostic, the pipeline-aware reward design is a sensible departure from failure-only prioritization, and the CI replay methodology avoids harming production. The open dataset is a concrete asset for the regression-test-optimization community, and the explicit treatment of post-submit flakiness as a transition-detection problem addresses a real gap between academic benchmarks and industrial practice. However, the strength of the empirical support is currently limited by the self-referential definition of relevant transitions in RQ2 and by the absence of statistical tests for the RQ1 comparisons; these issues must be addressed before the headline numbers can be taken at face value.","major_comments":[{"comment":"The three-CI-cycle flaky-transition threshold is used both to define the CostChangeRank reward labels during training and to construct the evaluation ground truth for the RQ2 claims. Because the threshold is selected from the churn rate of the same dataset, the reported 87% and 99.78% figures measure performance against a self-generated target rather than an externally validated notion of developer relevance. A flaky test that stabilizes only after four or more cycles would be labeled relevant, and a real regression fixed within three cycles would be labeled flaky, so both directions of misclassification are possible. Please provide a robustness analysis varying the threshold, or an external validation (e.g., developer-confirmed regressions and bug fixes), to demonstrate that the numbers are not an artifact of label construction.","section":"Section IV.C and Eq. (2), Figures 7 and 8"},{"comment":"The claim that PR-DQL 'outperforms existing approaches' is not supported by any significance testing. In Table V, PR-DQL and COLEMAN both report NAPFD 0.71 at an 80% budget, and at a 50% budget the difference is 0.71 versus 0.68 with standard deviations around 0.31; ROCKET and RANDOM are clearly worse, but the PR-DQL-versus-COLEMAN advantage is within one standard deviation. Reporting only means and standard deviations is insufficient to justify 'consistently achieving the highest NAPFD values.' Paired significance tests (e.g., Wilcoxon signed-rank) and effect sizes, or confidence intervals, are needed to support the comparative conclusion.","section":"Tables IV and V, Section V.A"},{"comment":"The ROCKET baseline was implemented by the authors because the original replication package is missing, but no validation of this reimplementation is provided. An unvalidated baseline implementation can bias the comparison, especially because ROCKET's heuristic involves several configurable weights. Please make the reimplementation publicly available and validate it against published ROCKET results on a benchmark dataset, or at least report the exact configuration used. In addition, RQ2 compares only against RANDOM and the authors' own RNChange variant; no established transition-based selection method, such as the Google approach cited as reference [35], is included, which weakens the claim of outperforming existing techniques in the post-submit setting.","section":"Section IV.A and Section V.B"},{"comment":"The headline claim that PR-DQL 'scheduled the first failing test within the first 16% of tests' is inconsistent with Table IV, where PR-DQL's NFR is reported as 0.17 ± 0.22. Table III reports 0.16 for CostRank, but the abstract and the conclusion attribute 16% to the full PR-DQL approach. Please reconcile whether the 16% figure refers to the reward-function comparison in Table III or to the final PR-DQL configuration in Table IV, and correct the abstract and conclusion accordingly.","section":"Abstract and Section V.A"}],"minor_comments":[{"comment":"The sentence 'there not only a need but also potential for further improvements' is missing a verb and should read 'there is not only a need but also potential for further improvements.'","section":"Abstract"},{"comment":"The y-axis label 'Ratio of Detected Transitions per CI Cycle [%]' is ambiguous; the surrounding text describes the percentage of executed test cases that reveal a relevant transition, so the axis label should be reworded to match the text.","section":"Figure 7"},{"comment":"The subsection heading 'Per-submit pipelines' appears to be a typo for 'Pre-submit pipelines'; the text immediately following it discusses pre-submit testing and the CostRank function.","section":"Section III.C"},{"comment":"The statement 'We use the publicly available open-source implementations of COLEMAN' is not accompanied by version or commit information; please include exact versions for reproducibility, particularly since the dataset is provided openly.","section":"Section IV.A"},{"comment":"The paragraph on safeguard mechanisms correctly acknowledges the risk of incorrect omissions, but it should be tied to the evaluation by stating whether any gate failures or incorrect culprit assignments were observed during the 20-week study period; as written, the deployment risk is acknowledged but not quantified.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and practically important problem, and the open dataset is a genuine contribution. The main risk is the circularity between the flaky-transition threshold used for training and for evaluation in RQ2; this needs to be addressed with a robustness analysis or external validation before the post-submit claims can be accepted. The RQ1 comparison also needs significance testing because some reported differences against COLEMAN are small relative to the standard deviations. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You can read this as a solid industrial case study. The authors built a Deep-Q-Learning test prioritizer for BMW's CI, using only test names and historical results as features, and they release the dataset. The genuinely new bits are the two pipeline-specific reward functions: CostRank for pre-submit (rewarding early failures while accounting for execution time) and CostChangeRank for post-submit (rewarding non-flaky transitions and penalizing flaky ones). That's a sensible distinction that most of the literature ignores.\n\nThe pre-submit results are reasonable: PR-DQL beats RANDOM and ROCKET clearly, and is a bit better than COLEMAN (NAPFD 0.75 vs 0.72). No significance tests, and the margin over COLEMAN is modest, but the direction is consistent across budgets. The post-submit story is where I'd be careful. The 87% and 99.78% numbers are computed against a 'relevant transition' label that is defined as a transition that does not switch back within three CI cycles (Section IV.C). That same three-cycle rule is baked into the CostChangeRank reward. So the evaluation measures how well the model optimizes the authors' own operationalization of relevance, not an externally validated notion. They give a rationale for the threshold (churn rate, developer resolution time), and it may be perfectly reasonable, but nothing in the paper checks it against developer-confirmed regressions or varies the threshold to show sensitivity. This is a construct validity issue rather than a fatal flaw, but it means the post-submit headline claims are conditional.\n\nOther soft spots: ROCKET was reimplemented by the authors because the original package is missing, which risks bias; the pre-submit dataset only includes failed cycles, which is standard in this literature but limits generalizability; and there are no statistical significance tests anywhere. The paper is honest about deployment safeguards in Section VI, which I appreciate.\n\nWho is this for? Researchers and practitioners working on regression test optimization for large-scale CI. It's worth a serious referee, and I'd want to see the reviewer push for sensitivity analysis on the flakiness threshold and more rigorous comparison to COLEMAN. I'd cite it for the dataset and the pipeline-aware reward design.","headline":"Solid industrial RL test-optimization paper with useful pipeline-aware rewards, but post-submit headline numbers hinge on a self-defined flakiness threshold.","tokens_in":16420,"tokens_out":3501,"would_cite":true,"duration_ms":31667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline-aware Deep-Q-Learning approach schedules the first failing test within the first 16% of the test suite and catches 87% of relevant transitions at half the budget.","keywords":["regression test prioritization","regression test selection","Continuous Integration","Deep Q-Learning","Reinforcement Learning","test flakiness","pipeline-aware","industrial case study"],"falsifier":"Re-run the post-submit evaluation on the same dataset while changing the flaky-transition threshold from three cycles to one cycle and to five cycles; if the reported 87% and 99.78% detection rates change by more than a few percentage points, the central result is an artifact of the chosen threshold. A second check is to inspect whether tests that flip twice within three cycles but then stabilize and stay flipped are nevertheless labeled flaky, which would mean genuine regressions are being discarded by the reward.","tokens_in":15386,"feed_emoji":"🧪","tokens_out":6889,"duration_ms":62491,"temperature":0.7,"pith_summary":"This paper argues that regression test optimization for large-scale continuous integration can be made practical by letting a Deep-Q-Learning agent learn pipeline-specific notions of which tests matter, using only lightweight, language-agnostic features. For pre-submit pipelines, where a failure blocks a merge, the paper defines a cost-aware reward (CostRank) and reports that the first failing test is scheduled within the first 16% of tests on average, beating a state-of-the-art bandit baseline and a heuristic baseline. For post-submit pipelines, where developers care about tests whose outcome changes between builds, the paper defines a transition-aware reward (CostChangeRank) that penalizes flaky transitions, and reports that 87% of relevant transitions are selected within half the test execution budget and 99.78% within five CI cycles. If these results hold, the approach would cut CI resource use and feedback latency in monorepo settings without needing per-test coverage data.","feed_headline":"CI optimizer finds first failing test in 16% of tests","feed_subtitle":"A Deep-Q-Learning prioritizer also catches 87% of relevant post-submit transitions at half the execution budget.","key_machinery":"The mechanism that carries the argument is a Deep-Q-Network agent that ranks each test target independently (pointwise ranking) and a pair of pipeline-specific reward functions. The pre-submit reward, CostRank, returns a positive value that increases the earlier a failing test appears and the shorter its execution time, and a negative value for passing tests, so the agent learns to surface likely failures quickly. The post-submit reward, CostChangeRank, returns 1 for a relevant transition, −1 for a flaky transition defined as one that flips back within three consecutive CI cycles, and the negative normalized duration otherwise, so the agent learns to prefer short, information-rich tests while avoiding flaky ones. Input features are limited to the last 25 test verdicts, a PCA-reduced bag-of-words encoding of the test target name, cycles since last failure, cycles since last execution, and average execution duration. The agent is trained with experience replay and an ε-greedy-style Gaussian exploration on action values.","core_discovery":"The central claim is that a single Deep-Q-Learning framework, trained on historical test results and test names, can outperform dedicated regression test optimization baselines in both pre-submit and post-submit industrial CI pipelines when the reward function matches the pipeline's goal. In pre-submit pipelines, where the goal is fast failure feedback, the authors define CostRank, which rewards failing tests more when they are scheduled earlier and when their execution cost is low; with this reward the approach reaches a mean NAPFD of 0.75 at a full budget and 0.53 even at a 10% budget, with the first failure appearing after 16% of the test suite on average. In post-submit pipelines, where the goal is to detect non-flaky pass-to-fail or fail-to-pass transitions, the authors define CostChangeRank, which rewards relevant transitions and assigns −1 to transitions that flip again within three CI cycles; with this reward the approach selects 87% of relevant transitions within half the execution budget and detects 99.78% within five cycles. The paper further shows that the cost-aware rewards outperform their non-cost-aware counterparts (RNFail and RNChange) and random ordering.","pith_inferences":["The three-cycle flakiness threshold is a business parameter, not a law of nature; a deployment with a faster or slower merge cadence should recalibrate it, and the paper's headline post-submit numbers would likely shift if recalibrated.","The pipeline-aware reward design transfers beyond CI to other gating loops, such as choosing which system-level tests to run before promoting a release candidate, where the same pass-to-fail/fail-to-pass distinction applies.","The 99.78% five-cycle figure is a recovery guarantee with delay tolerance, not instant detection; teams that need to revert a regression before several merges accumulate must budget for that latency or require a dedicated fail-fast stage.","Combining the test-name embedding with code-change signals (e.g., which files a PR touches) would likely sharpen the ranking further, but that extension is not explored in the paper."],"forward_implications":["Pre-submit pipelines can shorten developer feedback loops: with a 50% budget the first failing test appears at 16% of the suite on average, and even at a 10% budget the NAPFD stays at 0.53.","Post-submit pipelines can halve their resource consumption while still catching 87% of developer-relevant test transitions immediately and over 98% within three CI cycles.","Because the features are just historical verdicts and test names, the technique can be dropped into a multi-language monorepo without instrumentation or coverage harvesting.","The agent keeps learning from new execution results, so it tracks a CI environment where tests are added, renamed, and removed daily without manual retraining.","The cost-aware rewards generalize: any pipeline whose objective is fast failure feedback can reuse CostRank, and any pipeline where outcome changes matter can reuse CostChangeRank."],"supporting_citations":[{"why":"Supplies the Google-scale context showing why coverage-based features are infeasible and why flakiness undermines regression test optimization; the paper's motivation and feature design respond to this.","marker":"[5]"},{"why":"Establishes that pre-submit and post-submit CI stages need different test selection priorities, which is the premise for the pipeline-aware reward functions.","marker":"[17]"},{"why":"Provides COLEMAN, the multi-armed bandit baseline that PR-DQL is compared against in pre-submit prioritization.","marker":"[27]"},{"why":"Provides RETECS, the RL test selection approach whose exploration strategy and RNFail reward PR-DQL builds on, and the RNFail baseline in the pre-submit comparison.","marker":"[28]"},{"why":"Provides ROCKET, the heuristic failure-history baseline used in the pre-submit comparison.","marker":"[33]"},{"why":"Provides the industry practice of treating rapidly switching transitions as flaky, which directly informs the three-cycle flaky definition in CostChangeRank.","marker":"[35]"}],"fun_headline_variants":["RL test prioritizer finds first failure in 16% of suite","CI optimizer catches 87% of relevant transitions at half budget","Pipeline-aware RL optimizer improves CI test prioritization","Lightweight RL test prioritizer works across languages","RL approach cuts CI test budget while boosting failure detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's post-submit numbers depend on defining a flaky transition as one that flips again within three consecutive CI cycles; that same definition is used both as the reward signal during training and as the ground truth for evaluating whether a detected transition is relevant, so if a different threshold better matches what developers actually care about, the reported 87% and 99.78% figures would not be measuring that.","fun_headline_variants_meta":{"raw":{"variants":["RL test prioritizer finds first failure in 16% of suite","CI optimizer catches 87% of relevant transitions at half budget","Pipeline-aware RL optimizer improves CI test prioritization","Lightweight RL test prioritizer works across languages","RL approach cuts CI test budget while boosting failure detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3587,"prompt_tokens":1086,"completion_tokens":2501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":2422}},"tokens_in":702,"tokens_out":2501,"duration_ms":19509,"temperature":1.0,"reasoning_tokens":2422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:07:36.212895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the post-submit evaluation on the same dataset while changing the flaky-transition threshold from three cycles to one cycle and to five cycles; if the reported 87% and 99.78% detection rates change by more than a few percentage points, the central result is an artifact of the chosen threshold. A second check is to inspect whether tests that flip twice within three cycles but then stabilize and stay flipped are nevertheless labeled flaky, which would mean genuine regressions are being discarded by the reward.","supporting_citations":[{"cited_title":"Taming google-scale continuous testing,","cited_arxiv_id":null,"evidence_quote":"Supplies the Google-scale context showing why coverage-based features are infeasible and why flakiness undermines regression test optimization; the paper's motivation and feature design respond to this."},{"cited_title":"Techniques for improving re- gression testing in continuous integration development environments,","cited_arxiv_id":null,"evidence_quote":"Establishes that pre-submit and post-submit CI stages need different test selection priorities, which is the premise for the pipeline-aware reward functions."},{"cited_title":"A multi-armed bandit approach for test case prioritization in continuous integration environments,","cited_arxiv_id":null,"evidence_quote":"Provides COLEMAN, the multi-armed bandit baseline that PR-DQL is compared against in pre-submit prioritization."},{"cited_title":"Reinforcement learning for automatic test case prioritization and selection in continuous integration,","cited_arxiv_id":null,"evidence_quote":"Provides RETECS, the RL test selection approach whose exploration strategy and RNFail reward PR-DQL builds on, and the RNFail baseline in the pre-submit comparison."},{"cited_title":"Test case prioritization for continuous regression testing: An industrial case study,","cited_arxiv_id":null,"evidence_quote":"Provides ROCKET, the heuristic failure-history baseline used in the pre-submit comparison."},{"cited_title":"Assessing transition-based test selection algorithms at google,","cited_arxiv_id":null,"evidence_quote":"Provides the industry practice of treating rapidly switching transitions as flaky, which directly informs the three-cycle flaky definition in CostChangeRank."}],"review_version":1}