{"id":"ad89daef-ba2f-4e77-a40d-3a9f58230c01","arxiv_id":"2507.12483","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic survey of 115 reinforcement learning for software engineering papers finds most activity in quality assurance and test generation, with sparse coverage of requirements, management, and advanced RL algorithms.","lead":"This paper maps 115 studies that apply reinforcement learning to software engineering tasks, and summarizes trends, algorithms, datasets, and evaluation practices. It is a reference for researchers who want to see where RL has been used across the software lifecycle and where the gaps are.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Search string and screening cannot support the 'comprehensive/no relevant studies missed' claim; the 'award' typo, omitted 'reward' and other RL keywords, and arXiv preprints counted as peer-reviewed undermine the 115-study population.","rationale":"The reader identified the flawed search string and the peer-review status issue as the load-bearing weakness. My independent reading of Section 4.2 and the reference list confirms that the search string omits \"reward\" and includes the typo \"award\", and that at least three arXiv preprints are counted among the 115 peer-reviewed studies. The survey's descriptive statistics are internally consistent and clearly presented, but they are computed from a retrieval whose completeness and purity are not established. A corrected search and an audit of the retained references could plausibly change the reported distributions, so the appropriate posture is conditional, exactly as the reader concluded. I do not see an internal inconsistency in the taxonomy or analysis that would require rejection; the central problem is the representativeness of the study population. If the authors fix the search string, document screening reliability, and remove or justify non-peer-reviewed entries, the survey could become a reliable reference. My concern does not move the verdict because the reader already assigned CONDITIONAL for essentially the same reason.","tokens_in":40197,"tokens_out":3306,"duration_ms":39791,"concrete_test":"Independently rerun the search on ACM DL, IEEE Xplore, and ScienceDirect for the 22 listed venues, 2015-May 2025, with a corrected boolean query: (\"reinforcement\" OR \"Q-learning\" OR \"Q-network\" OR \"reward\" OR \"policy gradient\" OR \"PPO\" OR \"actor-critic\" OR \"bandit\") in title/abstract/keywords. Compare the retrieved count and the number passing the Section 4.3 inclusion criteria against the reported 169 and 115. Then extract the reference list of the survey, identify every entry that is an arXiv-only preprint rather than a peer-reviewed venue publication, remove those from the 115, and recompute the RQ1-RQ4 percentages (venue distribution, SE activity shares, algorithm shares, replicability).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central claim is that it offers the first systematic mapping of RL-for-SE based on 115 peer-reviewed studies from 22 premier venues. That claim depends on the retrieval and screening process recovering the relevant population. Section 4.2 defines the search string as (\"reinforcement\" OR \"Q-learning\" or \"Q-Network\" or \"award\") over titles, keywords, and abstracts in ACM DL, IEEE Xplore, and ScienceDirect. This string omits \"reward\", the most central term in RL abstracts, as well as common algorithm names such as \"policy gradient\", \"PPO\", \"actor-critic\", and \"bandit\". The inclusion of \"award\" appears to be a typo for \"reward\"; if so, the intended term is misspelled and will not match reward-based papers. If the string is taken literally, it will match metadata containing \"award\" (e.g., best paper awards) and introduce noise. Either way, the retrieval is both incomplete and contaminated. The snowballing step in Section 4.3 reports that no additional papers were found, which is implausible given these omissions and provides no independent verification that relevant studies were not missed. At least three retained references are arXiv preprints rather than peer-reviewed venue publications: Kim et al. (2024, arXiv:2411.07098), Sanchez-Stern et al. (2024, arXiv:2408.09237), and Wang et al. (2024, arXiv:2407.19487). Their inclusion contradicts the abstract's \"115 peer-reviewed studies\" and the stated venue-based inclusion criteria. Section 4.3 also reports abstract screening by two authors but gives no inter-rater agreement statistic, so screening reliability is unverifiable. Because every RQ1-RQ4 distribution is computed from this 115-paper set, a biased population would shift the reported percentages for venues, SE activities, algorithms, datasets, and replicability, directly undermining the survey's central quantitative map.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a systematic mapping study of reinforcement learning applied to software engineering. It claims to be the first systematic map of RL-for-SE, based on 115 peer-reviewed studies from 22 premier SE venues between 2015 and May 2025. The authors formulate five research questions and report quantitative distributions: 72% of studies address software quality assurance, 74% address generation tasks, Q-learning and DQN are the most used algorithms, and 41% of studies are non-replicable. The paper also identifies challenges and opportunities for RL-for-SE. The central claim depends on a supposedly comprehensive retrieval and screening procedure described in Sections 4.2 and 4.3.","tokens_in":40515,"tokens_out":3844,"duration_ms":47446,"significance":"If the underlying population of studies is correctly identified, the survey would be a useful and timely reference: it provides a structured taxonomy of SE activities, task types, and RL algorithms, along with trend data and a public artifact repository. The paper gives detailed tables (e.g., Table 6) and explicit quantitative summaries that are internally consistent with its own tables (e.g., 85/115 generation = 74%). The authors also make a welcome effort to report replicability rates and to discuss evaluation practices. However, the significance of the contribution is conditional on the completeness and purity of the 115-paper population, and the retrieval and screening methodology has load-bearing weaknesses that prevent accepting the current version as a comprehensive systematic map.","major_comments":[{"comment":"The search string is defined as (“reinforcement” OR “Q-learning” or “Q-Network” or “award”) over titles, keywords, and abstracts. This string omits the RL core term “reward” and common algorithm terms such as “policy gradient”, “PPO”, “actor-critic”, and “bandit”. If “award” is a typo for “reward”, the intended term is misspelled and will not match; if it is taken literally, it adds irrelevant matches and still misses reward-based papers. Section 4.3 then states that snowballing yielded no additional papers and concludes that no relevant studies were missed, but snowballing only inspects reference lists of already selected papers and cannot recover papers omitted by the flawed initial search. This undermines every RQ1–RQ4 distribution and the abstract's claim of a comprehensive map. The search must be rerun with an expanded, correctly spelled keyword set and the screening repeated, with the reported 115-count and all derived percentages recomputed.","section":"§4.2 and §4.3"},{"comment":"The abstract and Section 1 describe the surveyed set as “115 peer-reviewed studies published across 22 premier SE venues”, and Section 4.3 explicitly excludes “non-published manuscripts”. However, at least three retained references are arXiv preprints rather than venue publications: Kim et al. [60] (arXiv:2411.07098), Sanchez-Stern et al. [105] (arXiv:2408.09237), and Wang et al. [137] (arXiv:2407.19487). These papers appear in the analyzed corpus (e.g., [60] is listed in Table 6 under Multi-Agent Q-Learning) and their inclusion contradicts the stated inclusion criteria. The authors should replace these with their peer-reviewed versions if they exist, or explicitly reclassify the corpus. This materially affects the claim of surveying only peer-reviewed venue publications.","section":"§4.3 and references [60], [105], [137]"},{"comment":"The screening process is described as a manual review of abstracts by two authors, but no inter-rater agreement metric (e.g., Cohen's kappa) is reported and no explicit procedure for resolving screening disagreements is given. Since the selection of the 115 papers is the foundation of all subsequent statistics, the absence of reliability information weakens the claim that the screening is reproducible and that papers were not arbitrarily excluded. The authors should report screening agreement and conflict resolution, and ideally apply the inclusion criteria to full texts where abstracts are ambiguous.","section":"§4.3"}],"minor_comments":[{"comment":"Row 5 appears to be a typographical merge: “International Symposium on Testing and Analysis Working Conference on Mining Software Repositories” should be two separate entries, ISSTA and MSR. Also, “Information and Software Systems” should likely be “Information and Software Technology”, and “Automated Engineering” should likely be “Automated Software Engineering”.","section":"Table 2"},{"comment":"The abbreviation “TPRO” is used for Trust Region Policy Optimization; the standard abbreviation is TRPO, and the text elsewhere uses TRPO-context terminology.","section":"Table 6"},{"comment":"The text says “The cumulative plot in Figure 4 also illustrates…”, but the cumulative plot appears in Figure 3(b); Figure 4 instead shows venue distributions.","section":"§5, Figure 4"},{"comment":"There are minor typos: “alogorithms” in Section 1 and “Saftey Improvement” in Figure 5 should be “algorithms” and “Safety Improvement”, respectively.","section":"§1 and Figure 5"},{"comment":"The sentence “journal papers exhibit lower applicability” should read “lower replicability”, matching the subsequent discussion and Table 9.","section":"§8, Replicability paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear structure and a useful taxonomy, but the completeness claim is not supported by the reported search and screening. The fixes required—rerunning the search with a corrected keyword set, re-screening, and reconciling the arXiv preprints—are substantial but bounded tasks. If the authors can demonstrate that the corrected population still supports the main distributions, the survey would be publishable; in its current form, the central claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this one is a genuinely useful first pass at mapping RL across the whole SE lifecycle, not just testing. It updates the window to May 2025, covers 115 studies across 22 premier venues, and the classification into SE activities and RL algorithm families is careful and clearly presented. The descriptive numbers (72% SQA, 74% generation, value-based dominance) are internally consistent with the tables, and the artifact link is live. If you need a starting point for where RL has been applied in SE, this gives you a decent map.\n\nThe soft spots are in the retrieval and screening, and they hit the central claim. Section 4.2's search string is (\"reinforcement\" OR \"Q-learning\" or \"Q-Network\" or \"award\"). That \"award\" is a typo for \"reward\", so the string omits the single most common RL keyword and would match best-paper-award entries instead. It also misses \"policy gradient\", \"PPO\", \"actor-critic\", and \"bandit\". Given those omissions, the snowballing step finding zero additional papers is not credible. The inclusion of at least three arXiv preprints (Kim et al., Sanchez-Stern et al., Wang et al.) as \"peer-reviewed\" contradicts the stated criteria. And two-author abstract screening with no inter-rater agreement means the 115-paper set is assembled on faith. Section 10's threats-to-validity paragraph acknowledges search-term limitations but doesn't repair them.\n\nThat said, the distributions are probably not wildly off for the venues actually searched—the bias matters more for the comprehensiveness claim than for the rough shape of the field. The paper is still worth engaging with, but a referee should demand a corrected and rerun search, verification of publication status for every retained paper, and a reporting of inter-rater agreement.\n\nMy take: send it to review, but flag the methodology as needing a serious revision before acceptance. I'd cite it with a caveat and might bring it to reading group to discuss what \"comprehensive\" should mean in a mapping study.","headline":"Useful first systematic map of RL across the SE lifecycle, but the search string typo and arXiv-preprint miscounting undercut the comprehensiveness claim.","tokens_in":41130,"tokens_out":1558,"would_cite":true,"duration_ms":20401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey sets out to provide the first systematic map of reinforcement learning applied to software engineering, based on 115 peer-reviewed studies from 22 top venues since 2015.","keywords":["Software Engineering","Reinforcement Learning","Deep Reinforcement Learning","Systematic Literature Review","Systematic Mapping Study","Software Quality Assurance","Test Generation","Replicability"],"falsifier":"Re-run the same search over the same 22 venues with a corrected string that adds 'reward' and removes 'award', then apply the same inclusion criteria; if the corrected search retrieves additional relevant papers beyond the 115, the survey's claim that snowballing found no missed studies is contradicted.","tokens_in":39995,"feed_emoji":"🗺️","tokens_out":6476,"duration_ms":66494,"temperature":0.7,"pith_summary":"The paper sets out to give the field its first systematic map of where reinforcement learning has been applied in software engineering. It reviews 115 peer-reviewed papers from 22 premier software-engineering venues published since 2015, and reports that RL-for-SE is heavily concentrated in quality assurance (72 percent), especially test generation, with generation tasks overall at 74 percent. It also reports that Q-learning and DQN dominate algorithm choices, that most studies propose new techniques rather than run empirical comparisons, and that 41 percent of studies cannot be replicated from available artifacts. If the map is accurate, it gives researchers and practitioners a quantitative picture of an emerging field and a basis for deciding where to contribute.","feed_headline":"115 papers mapped: reinforcement learning in software engineering","feed_subtitle":"Survey finds quality assurance dominates, Q-learning and DQN lead, and 41% of studies cannot be replicated.","key_machinery":"The carrying mechanism is the systematic mapping-study protocol: a fixed search string over title, keyword, and abstract fields in three major scholarly databases, restricted to 22 premier software-engineering venues, followed by two-author abstract screening, snowballing, and card-sorting-based keywording. This machinery turns a scattered literature into distributions over venues, SE activities, task types, algorithms, data sources, evaluation metrics, and replicability that answer the paper's five research questions.","core_discovery":"The central discovery is a quantitative landscape of RL-for-SE: among 115 studies, 72 percent target software quality assurance, and test generation alone accounts for 49 papers. Task-wise, 74 percent of the underlying SE tasks are generation tasks, while ranking, classification, and regression are rare. Value-based algorithms appear in 52 percent of papers, led by Q-learning (27 papers) and DQN (22 papers), with PPO the most common policy-based method (17 papers). Only 5.2 percent of papers focus solely on refining core RL concepts, while 82.6 percent treat RL as a tool with task-specific tuning. Evaluation is effectiveness-heavy, and 41 percent of papers are non-replicable because source code or complete packages are missing. The paper frames these results as the first systematic mapping covering the whole software-engineering lifecycle, in contrast to earlier reviews limited to testing or to machine learning and deep learning broadly.","pith_inferences":["Because the search string omits the term 'reward', the reported 115-paper corpus likely undercounts studies that describe their learning signal without using the word 'reinforcement'; a corrected search could shift the activity and algorithm distributions.","The restriction to 22 premier venues and papers of at least 8 pages means the sharp post-2022 growth curve probably underestimates the total volume of RL-for-SE work appearing in workshops, short papers, and adjacent venues.","The survey's own distributions imply a concrete next step it does not take: assembling a shared benchmark environment suite for test generation and repair, since a 41 percent non-replicability rate makes cross-study comparison currently unreliable."],"forward_implications":["Software quality assurance is by far the most explored activity (72 percent of 115 papers), with test generation alone accounting for 49 papers, so other lifecycle activities such as requirements and management have almost no RL work.","Because 74 percent of RL-for-SE tasks are generation tasks, RL is currently used mainly to produce artifacts like tests, code, and comments rather than for classification, ranking, or regression.","Q-learning and DQN dominate the algorithm landscape, while more recent methods such as SAC, TD3, meta-RL, and RLHF appear in only a handful of studies, indicating room for the field to adopt newer RL techniques.","Only 5.2 percent of papers innovate on core RL concepts, and 41 percent are non-replicable, which together limit cumulative progress and fair comparison across studies.","Integration of LLMs with RL has emerged since 2024 in about 10 of the surveyed studies, pointing to a rapidly growing direction for RL-for-SE."],"supporting_citations":[{"why":"Supplies the systematic mapping-study guidelines for search, screening, and keywording that structure the survey.","marker":"[95]"},{"why":"Provides the comparative-analysis protocol used to position this review against prior systematic reviews.","marker":"[94]"},{"why":"Serves as the main comparison baseline for a deep-learning-for-SE review and supplies the data-source categories and contribution-type classification.","marker":"[146]"},{"why":"Provides the broader machine-learning and deep-learning-for-SE review that this survey extends by focusing specifically on reinforcement learning.","marker":"[134]"},{"why":"Is the only prior RL-specific review, limited to software testing, and is the scope this survey broadens to all of software engineering.","marker":"[2]"},{"why":"Supplies the software-engineering body-of-knowledge taxonomy of six SE activities used for categorizing the surveyed studies.","marker":"[12]"},{"why":"Defines the 2015 starting point of the deep reinforcement learning era that sets the survey's time interval.","marker":"[84]"},{"why":"Provides the standard RL background on Markov decision processes, policies, and value functions used throughout the survey.","marker":"[120]"},{"why":"Defines Q-learning, the algorithm most frequently identified in the surveyed RL-for-SE studies.","marker":"[139]"}],"fun_headline_variants":["115 studies map RL across software engineering","RL for software engineering: 115 studies surveyed","First RL4SE survey: 115 papers, QA dominates","Q-learning and DQN lead 115 RL4SE papers","41% of RL4SE studies lack replicability: survey"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's entire map depends on the assumption that its search and screening procedure found all relevant RL-for-SE papers in the chosen venues; the search string omits the common term 'reward' and instead contains the misspelled term 'award', so papers that only mention reward-based learning without the word 'reinforcement' in title, keywords, or abstract could be missing from the 115.","fun_headline_variants_meta":{"raw":{"variants":["115 studies map RL across software engineering","RL for software engineering: 115 studies surveyed","First RL4SE survey: 115 papers, QA dominates","Q-learning and DQN lead 115 RL4SE papers","41% of RL4SE studies lack replicability: survey"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3467,"prompt_tokens":981,"completion_tokens":2486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2407}},"tokens_in":597,"tokens_out":2486,"duration_ms":18889,"temperature":1.0,"reasoning_tokens":2407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:34:37.009790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same search over the same 22 venues with a corrected string that adds 'reward' and removes 'award', then apply the same inclusion criteria; if the corrected search retrieves additional relevant papers beyond the 115, the survey's claim that snowballing found no missed studies is contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the systematic mapping-study guidelines for search, screening, and keywording that structure the survey."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the comparative-analysis protocol used to position this review against prior systematic reviews."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the main comparison baseline for a deep-learning-for-SE review and supplies the data-source categories and contribution-type classification."},{"cited_title":"Sutton and Andrew G","cited_arxiv_id":null,"evidence_quote":"Provides the standard RL background on Markov decision processes, policies, and value functions used throughout the survey."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Q-learning, the algorithm most frequently identified in the surveyed RL-for-SE studies."}],"review_version":1}