{"id":"f3385e5d-7912-476e-a8cc-f5a1d039d77c","arxiv_id":"2504.18985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A seven-function longitudinal industrial case study reports LLM test generation scores improving from 32.44% to over 91% in nine months, but the improvement is partly driven by iterative prompt engineering on the same evaluation set.","lead":"A Spanish software consultancy built and tested a scoring framework that repeatedly measures how well LLM tools like GitHub Copilot generate unit and integration tests, tracking metrics such as code coverage, compilation success, and expert review. The study reports weighted scores rising from about 32% in March 2024 to more than 90% for the best models in December 2024, but the gains mix model improvements with prompt and metric tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 32.44% to 90% trajectory in Table 5 conflates prompt refinement, model identity, and calendar time, so it does not isolate LLM capability evolution.","rationale":"The reader's weakest assumption matches the main threat to the central claim. Additional inspection of Table 5 shows the comparison is not a same-model time series: the 32.44% row is ChatGPT-4 with the first prompt, the 67.96% row is ChatGPT-4 with the iterated prompt, and the 90% rows are o1-Preview and Claude 3.5 Sonnet. The paper even reports o1-Mini at 63.81%, lower than the May 2024 ChatGPT-4 value, so 'newer models are better' is not monotonic. Because the advertised result is capability evolution, this confound is load-bearing. However, the framework's practical contribution, including metric design, tool integration, and the replication package, plus the authors' explicit validity discussion, is credible. A conditional acceptance requiring a prompt-constant replication is the appropriate outcome, so the reader's verdict stands unchanged.","tokens_in":14404,"tokens_out":4702,"duration_ms":47483,"concrete_test":"Run a 2x2 controlled comparison on at least three of the seven functions: (a) initial prompt on the March 2024 model, (b) final optimized prompt on the March 2024 model, (c) initial prompt on a December 2024 model, and (d) final prompt on a December 2024 model. If (b) is close to (d) while (c) is close to (a), the prompt factor dominates and the capability-evolution narrative fails; if (c) substantially exceeds (a) under the identical initial prompt, model evolution is supported. If the exact March 2024 Copilot model is unavailable, approximate it with GPT-4 (e.g., gpt-4-0613) and compare against the newest model through the same API, using the exact prompts from the Zenodo replication package.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline evidence for 'rapid evolution of LLM capabilities' is confounded. The March-to-May ChatGPT-4 improvement (32.44% to 67.96%) is exactly the period during which Section 5.3 and Section 5.4 describe iterative prompt refinement, and the December 'over 90%' rows come from different models (o1-Preview, Claude 3.5 Sonnet) rather than from the same model tracked over time. Section 8's Internal Validity admits that iterative prompt refinement might confound the results, and the stated mitigation ('maintain consistent base prompts') is difficult to reconcile with Section 5.4's detailed evolution into a final prompt with six new instruction areas. The data are therefore equally consistent with prompt engineering, not model evolution, being the main driver. The framework itself remains useful, but the central evolutionary claim is not cleanly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for continuously evaluating commercial LLM-based test generators in industrial settings, combining objective metrics (code quality, white-box coverage) with subjective expert-assessed black-box metrics into a weighted composite score. The framework is instantiated at LKS Next, where seven functions from internal projects are used as subjects, expert-written tests serve as ground truth, and prompts are iteratively refined. A longitudinal study from March 2024 to December 2024 reports a rise in the weighted score from 32.44% (ChatGPT-4, March) to over 90% (o1-Preview and Claude 3.5 Sonnet, December), which the paper interprets as evidence of rapid evolution of LLM test-generation capabilities. The paper also discusses industrial adoption factors such as cost, integration, data privacy, and expert oversight.","tokens_in":14542,"tokens_out":2654,"duration_ms":27998,"significance":"The framework addresses a real, under-served need: industrial practitioners lack systematic, repeatable methods for re-evaluating LLM test generators as models and tools change. The paper's strengths include a concrete metric taxonomy, integration with widely used tools (SonarQube, JaCoCo, JUnit, Maven), a replication package, and an unusually candid threats-to-validity section that acknowledges prompt-refinement confounding. If the methodological issues below are resolved, the framework could serve as a template for other organizations and as a baseline for future longitudinal studies. The paper also adds useful qualitative evidence on practical adoption hurdles, such as the need for prompt engineering, training, and expert oversight. However, as written, the central empirical claim of 'rapid evolution of LLM capabilities' is not cleanly supported by the data, because the observed changes conflate three variables: calendar time, model identity, and the authors' own iterative prompt refinement.","major_comments":[{"comment":"The paper's headline result—scores rising from 32.44% to over 90%—is presented as evidence of 'rapid evolution of LLM capabilities,' but the comparison conflates prompt refinement with model evolution. The March-to-May improvement for ChatGPT-4 (32.44% to 67.96%) occurred during the exact period in which §5.3 and §5.4 describe iterative prompt refinement, which added six new instruction areas, including an explicit mandate to 'need a 100% of line, condition and condition/decision coverage.' The Section 8 mitigation that 'maintain consistent base prompts' is difficult to reconcile with §5.4's detailed prompt evolution. Please re-analyze the data separating prompt version from model version, for example by running the March prompt on the May model and the May prompt on the March model, or by reporting the prompt version used for each column of Table 5. Without such a separation, the improvement cannot be attributed to LLM capability evolution.","section":"§6, Table 5; §5.3–5.4; §8 Internal Validity"},{"comment":"The December 'over 90%' rows come from different models (o1-Preview, Claude 3.5 Sonnet), not from the same model tracked over time. The abstract and conclusions state that 'scores rising from 32.44% ... to over 90%' demonstrate 'the rapid evolution of LLM capabilities,' but this trajectory mixes model identity, calendar time, and prompt version. The data are equally consistent with the interpretation that newer models simply perform better at this task, or that the authors' prompt engineering transferred to new models. If the claim is about the ecosystem evolving, the text should say so explicitly and not imply that a single model's capabilities improved. Please either restrict the longitudinal claim to same-model comparisons (e.g., ChatGPT-4 March vs. ChatGPT-4 May) or reframe the conclusion as an evaluation of the current state of multiple commercial models.","section":"§6, Table 5; §10"},{"comment":"The scoring formulas for penalty metrics CE and SAI normalize by MAX(CE value for all LLMs) and MAX(SAI value for all LLMs). This makes each LLM's score depend on the cohort of models included in the evaluation, not on an absolute standard. When a new model with worse compilation errors is added to the comparison, the MAX increases and every other model's penalty decreases, so the composite scores from different evaluation waves are not directly comparable. Since Table 5 compares March, May, and December evaluations with different model cohorts, the reported total scores are not on a common scale. Please either fix the normalization to an absolute reference (e.g., a fixed maximum or the expert baseline) or demonstrate that the conclusions are robust to the normalization choice. This is load-bearing for the longitudinal claim.","section":"§4.4, Table 4"},{"comment":"The paper reports a single value per metric per model per time point, with no repeated runs, no variance measures, and no statistical tests. Section 8 states that 'validate significant changes in performance through multiple test runs' is a mitigation, but the results section reports no such replications, and the methodology section does not specify the number of runs or how sampling was handled. Given the known nondeterminism of LLM generation, a single run per configuration is insufficient to support the precise percentage differences (e.g., 98.00% vs. 95.71% line coverage, differences of 2–6 percentage points in several black-box metrics). Please add replication details and error bars, or explicitly state the run counts and any observed variability; if only one run was performed, temper the precision of the claims accordingly.","section":"§8 Internal Validity; §6"}],"minor_comments":[{"comment":"The word 'reseaarch' is a typo; it should read 'research.'","section":"§3"},{"comment":"The abbreviation 'BV A' appears with an extra space; it should be 'BVA.' The same issue recurs in the list of black-box metrics.","section":"§4.4"},{"comment":"The paper variously refers to 'GitHub Copilot' and 'ChatGPT-4' as the subject of the early evaluations. Clarify the relationship: if GitHub Copilot used GPT-4 in March 2024, state explicitly how the 'ChatGPT-4' rows of Table 5 were obtained, and whether the same interface (Copilot vs. direct API/chat) was used across all evaluations.","section":"§1, §6"},{"comment":"The conclusions state the study ran 'From April 2024 to January 2025,' but the results section and Table 5 describe evaluations from March 2024 to December 2024. Please reconcile the reported time frame.","section":"§10 vs. §6"},{"comment":"The ACM reference template lines 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' are left in the manuscript and are irrelevant to the study; they should be removed or replaced with actual submission dates.","section":"Appendix/References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the EASE community as an industrial experience report. However, the editorial decision should weigh whether the authors can address the confounding between prompt refinement and model evolution within a revision; if they can add a prompt-controlled comparison or clearly restrict the claims, the paper may be publishable. The MAX-normalization issue in the scoring formula is a technical concern that should be fixed or discussed, as it affects the comparability of the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the paper's real value is the measurement framework itself, not the headline numbers. The authors define a weighted score combining code quality, white-box coverage, and expert-judged black-box metrics, integrate it with SonarQube in a real consultancy, and document a three-phase evaluation loop. That is a genuinely useful blueprint for industrial teams who need to re-evaluate commercial LLM tools as they update. They also make sensible choices on data leakage (functions without public tests) and provide a replication package.\n\nThe soft spot is the central claim that the 32% to 90% trajectory shows rapid LLM evolution. It does not, as presented. The March-to-May ChatGPT-4 improvement coincides with the prompt engineering described in Section 5.4, where they explicitly instruct the model to target 100% line/condition/decision coverage and to follow SonarQube rules. The December rows come from different models entirely. The Section 8 mitigation—'maintain consistent base prompts'—contradicts the detailed prompt evolution in Section 5.4. So the data are equally consistent with prompt fitting and model identity, not capability evolution. The authors admit this threat, which is to their credit, but they then undercut their own headline conclusion in the Discussion.\n\nOther soft spots are minor by comparison: seven functions from one company, no repeated runs, no error bars, and a weighted scheme that is admittedly arbitrary. The claim that the improvement 'persists across different weighting schemes' appears without showing those alternative weightings. The expert ground-truth tests are also used as both the benchmark and the basis for black-box metric judgment, which is fine but should be stated more explicitly.\n\nNone of this sinks the paper. The framework is reusable, the writing is direct, and the threats-to-validity section is candid. I would send this to peer review with a request that the authors either separate prompt effects from model effects (e.g., stable prompt across versions) or reframe the contribution as a measurement method rather than evidence for LLM evolution. As it stands, the paper is a solid industrial case study with an overclaimed headline.\n\nI'd bring it to a reading group focused on LLM testing evaluation—there is a good discussion to be had about what counts as evidence in these longitudinal industrial studies. I would cite the framework, but not the evolution numbers.","headline":"The framework is a useful, honest industrial measurement contribution; the 'rapid evolution' headline is confounded with prompt refinement and model switching.","tokens_in":15074,"tokens_out":3035,"would_cite":true,"duration_ms":29353,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Commercial LLM test generators improved from a 32.44% weighted quality score in March 2024 to over 90% by December 2024, and a continuous evaluation framework is needed to track such fast-moving tools.","keywords":["Large Language Models","Test Generation","Continuous Evaluation","Industrial Case Study","DevSecOps","Code Coverage","Prompt Engineering"],"falsifier":"Re-run the full seven-function evaluation with the March 2024 prompt frozen, applying it to the December 2024 models. If the weighted scores stay near the March levels or rise only modestly, the improvement is mostly prompt engineering; if they still reach roughly the 90% range, the capability-evolution claim survives.","tokens_in":14202,"feed_emoji":"🧪","tokens_out":8271,"duration_ms":70972,"temperature":0.7,"pith_summary":"The paper is trying to establish that the rapid evolution of commercial LLM test generators makes point-in-time evaluations obsolete for industry, and that a continuous measurement framework can keep adoption decisions current. It builds such a framework around metrics that combine automated measurements (compilation success, static analysis, coverage, test isolation) with expert judgment (equivalence partitioning, boundary values, parameterization, and coverage of expert-written scenarios), then validates it in a nine-month longitudinal study at a mid-sized software consultancy. The weighted quality score rose from 32.44% for the first GPT-4 evaluation to over 90% for the best models by December 2024, while compilation errors per test suite dropped from 31 to near zero. The argument matters because DevSecOps teams must decide when to adopt these tools, which model to pick, and how much prompt investment is worthwhile, and any of those decisions based on a single snapshot will go stale within months.","feed_headline":"LLM test scores jumped from 32% to 90% in 2024","feed_subtitle":"A nine-month industrial study shows why point-in-time evaluations of coding AI go stale fast.","key_machinery":"The load-bearing mechanism is a weighted scoring framework that converts heterogeneous quality signals into a single comparable number over time. Objective metrics—compilation errors (−20%), static analysis issues (−5%), setup/teardown usage (+10%), and line, branch/decision coverage with test isolation (+40% combined)—are weighted 50% against subjective black-box metrics: equivalence partitioning, boundary value analysis, test parameterization, and expert-generated test coverage (the other 50%). Around this sits an iterative prompt-engineering loop that refines the prompt after each evaluation cycle, so the pipeline measures the model-plus-prompt system rather than the model alone; reproducibility comes from a documented toolchain and a fixed set of seven test subjects with expert ground truth.","core_discovery":"The central claim is that LLM test generation capabilities improved dramatically during 2024, and that this improvement can be tracked and managed only with a measurement process designed for repeated evaluation. Using seven carefully selected functions from projects without public test suites to avoid data leakage, expert QA engineers built ground-truth tests, and the framework scored LLM-generated tests across code quality, white-box, and black-box metric categories. The longitudinal results show GitHub Copilot's GPT-4 producing an average of 31 compilation errors per suite in March 2024, falling to 3 by May 2024 after iterative prompt refinement, while newer models reached near-perfect compilation, line and branch coverage above 95%, test parameterization at 88–92%, and total weighted assessment scores above 90%. The authors interpret these results as evidence that the technology is maturing toward industrial viability, while still requiring expert oversight.","pith_inferences":["I would caution that the headline 32% to 90% rise should be read as the improvement of the combined model-plus-prompt system, not of the models alone; a clean separation would require a frozen-prompt longitudinal run.","The framework's equal weighting of objective and subjective metrics is an organizational choice, not a universal law; the authors make this explicit, and a company that cares more about maintainability than coverage would likely reorder the ranking by changing weights.","A natural extension is to run the same seven test subjects and prompts against locally hosted or in-house models, which would directly test the privacy-motivated deployment path the paper names as future work.","The seven-function selection means the 90% figure is a tracking instrument for one consultancy's priorities, not a general benchmark; other organizations should expect different absolute numbers but likely the same steep slope."],"forward_implications":["Adoption decisions based on a single evaluation will routinely be wrong: a tool that looks unusable in March can clear a 90% weighted bar by December, so companies should schedule re-evaluations on the same timescale as model releases.","Prompt engineering is part of the measured system; GPT-4's score rose from 32.44% to 67.96% between March and May 2024 with the same underlying model, so organizations should track prompt versions as carefully as model versions.","The highest-scoring models reached expert-generated test coverage above 90%, suggesting generated tests can now capture most, though not all, of what expert testers write.","Cost and latency remain load-bearing: newer top-scoring models carry higher computational cost, and per-seat licensing matters for larger teams, so adoption is a portfolio decision rather than a pure quality ranking."],"supporting_citations":[{"why":"Survey of LLM-based software testing that frames the academic landscape and the gap in real-world evaluation that this continuous framework fills.","marker":"[Wang et al. 2024b]"},{"why":"Empirical study of JUnit test generation with earlier LLMs that provides the early-coverage baseline this longitudinal study extends.","marker":"[Siddiq et al. 2024]"},{"why":"TESTPILOT's demonstration that prompt engineering and iterative refinement improve generated tests, supporting the paper's own prompt-loop methodology.","marker":"[Schäfer et al. 2024]"},{"why":"Comparative assessment of ChatGPT against search-based testing that supplies a prior point-in-time reference for LLM test capability.","marker":"[Tang et al. 2024]"},{"why":"Documents inter-dataset code duplication and data leakage in LLMs, underpinning the paper's requirement to choose test subjects without pre-existing public test suites.","marker":"[López et al. 2025]"},{"why":"Introduces prompt chaining, the technique the paper adapts to structure its iterative prompts.","marker":"[Wu et al. 2022]"},{"why":"Shows that automatically generated tests often contain problematic assertions and structures, motivating the expert-assessment component of the framework.","marker":"[Fan 2019]"}],"fun_headline_variants":["LLM test scores jump from 32% to 90% in 2024","Point-in-time LLM test scores go stale fast, study finds","New framework keeps LLM test quality evaluations current","LLM test compilation errors drop from 31 to 3 in 2 months"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the measured improvement over time reflects real evolution in LLM capabilities rather than the authors' own iterative prompt refinement; the paper itself flags this as a confound, and if it gives way, the headline 32% to 90% trend becomes a prompt-engineering artifact.","fun_headline_variants_meta":{"raw":{"variants":["LLM test scores jump from 32% to 90% in 2024","Point-in-time LLM test scores go stale fast, study finds","New framework keeps LLM test quality evaluations current","LLM test compilation errors drop from 31 to 3 in 2 months"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3733,"prompt_tokens":930,"completion_tokens":2803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2725}},"tokens_in":546,"tokens_out":2803,"duration_ms":22270,"temperature":1.0,"reasoning_tokens":2725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:04:03.730661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full seven-function evaluation with the March 2024 prompt frozen, applying it to the December 2024 models. If the weighted scores stay near the March levels or rise only modestly, the improvement is mostly prompt engineering; if they still reach roughly the 90% range, the capability-evolution claim survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Empirical study of JUnit test generation with earlier LLMs that provides the early-coverage baseline this longitudinal study extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Comparative assessment of ChatGPT against search-based testing that supplies a prior point-in-time reference for LLM test capability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that automatically generated tests often contain problematic assertions and structures, motivating the expert-assessment component of the framework."}],"review_version":1}