{"id":"abb9df63-0d99-45e8-8891-927b322b140f","arxiv_id":"2411.17927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Regression-based analysis of CodeGen1-Multi across three software engineering tasks finds no evidence of emergent capabilities in the tested scale range.","lead":"This paper presents EMERGENTEVAL, a pipeline for detecting emergent capabilities in LLMs on software engineering tasks, and applies it to the CodeGen1-Multi model family. It finds no evidence that performance jumps emerge as model size grows from 350M to 16.1B parameters across bug fixing, code translation, and commit message generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detector sensitivity is unvalidated: with only four scales and a hand-set threshold, EmergentEval may fail to flag genuine emergence, so the negative finding is not yet supported.","rationale":"The reader's weakest assumption correctly identifies the regression threshold and detection power as the load-bearing premise. I agree: the abstract's claim is a negative existential statement about emergence in the selected tasks, and that claim requires the measurement pipeline to be sensitive enough to see emergence if it existed. The paper does not demonstrate this sensitivity. It uses four scales, a threshold t=0.10 described as 'established before testing' but without published calibration details, and two deviation metrics that can disagree on the same data (Table 2, prompt1). The RQ3 floor effect is a further instance: when all B-Moses scores are zero, the regression is trivially perfect and the pipeline concludes 'non-emergent' regardless of whether the model has any capability at all. These are not fatal flaws; the paper is honest in scoping its claim and in discussing threats. But they make the result conditional on a detector that has not been shown to work. The recommended check is a synthetic power analysis at the same scales and score ranges, which would directly test whether EmergentEval can detect a known emergent jump. If the detector cannot, the negative finding should be reported as 'the pipeline did not detect emergence' rather than 'no evidence of emergence.' This is consistent with the reader's conditional verdict, so I do not propose changing it.","tokens_in":14806,"tokens_out":3851,"duration_ms":36912,"concrete_test":"Run a simulation-based power analysis at the exact four scale values (0.35, 2.7, 6.1, 16.1). Generate synthetic metric scores with the same sample sizes and noise levels as the observed BLEU/CodeBLEU data under four scenarios: (1) linear growth with Gaussian noise; (2) a genuine emergent step-jump, e.g., a 0.3-point jump at 16.1B; (3) a smooth nonlinear but predictable power-law curve; and (4) an all-zero floor pattern as in RQ3. Repeat 500 times per scenario, apply EmergentEval with t=0.10 under both RMSD and MAE, and compute the detection rate. If scenario (2) is detected in fewer than 80% of runs, or scenario (1) is falsely flagged more than 5% of the time, the threshold and detector are miscalibrated, and the paper's negative finding cannot be separated from detector insensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that scaling CodeGen1-Multi from 350M to 16.1B parameters shows no evidence of emergent capabilities on three SE tasks. This negative conclusion is only meaningful if EmergentEval would reliably detect emergence when it is present. That condition is not established. The detector fits a linear regression through exactly four scale points and compares RMSD/MAE against a threshold t=0.10 chosen informally in Section IV.A. With four points, the residual pattern has very low power to distinguish a sharp, unpredictable jump from ordinary scatter around a line; a single high-leverage point can dominate the regression. The paper itself demonstrates threshold sensitivity in Table 2: prompt1 under RMSD exceeds t=0.10 while MAE does not, and the authors cannot decide whether this is negative emergence. Further, RQ3 results are at floor: B-Moses scores are exactly zero across all scales (Table 4), so p=0 is trivially below threshold and the pipeline reports 'non-emergent' even though the model never performs the task. Without a power analysis or calibration against known ground-truth emergence curves at these sample sizes and score ranges, the absence of evidence is indistinguishable from the detector's inability to see evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EmergentEval, a model-agnostic pipeline for detecting emergent capabilities in LLMs. The pipeline fits a linear regression to performance scores versus model scale (parameter count), computes residual statistics (RMSD and MAE) as a regression value p, and flags the curve as emergent when p exceeds a pre-set threshold t = 0.10. The authors instantiate the pipeline on the CodeGen1-Multi family (350M, 2.7B, 6.1B, 16.1B) across three software engineering tasks: bug fixing, code translation, and commit message generation, using several prompting strategies and metrics (EM, BLEU, CodeBLEU, B-Moses, B-Norm), including Checklist-based perturbed inputs. Their main empirical claim is that they find no evidence of emergent capabilities in the selected tasks over the tested scale range; performance tends to grow approximately linearly or not at all. The paper also contributes a replication package and positions the framework as a reusable evaluation tool for future emergence studies.","tokens_in":15019,"tokens_out":6082,"duration_ms":52908,"significance":"If the negative result holds up, the paper provides a useful empirical data point for the ongoing debate about emergent abilities, extending it to the software engineering domain. It would reassure practitioners that scaling CodeGen1-Multi within the 0.35B-16.1B range does not produce unpredictable capability jumps, countering the intuition that adding scale alone can unlock qualitatively new behavior. The explicit pipeline and replication package are valuable assets, and the use of continuous metrics plus perturbation analysis is methodologically sensible. However, the significance is conditional on the validity of the detection method, which the paper does not establish; the framework and findings are a promising starting point rather than a conclusive answer.","major_comments":[{"comment":"The central negative claim is only meaningful if EmergentEval would detect genuine emergence when present. The paper calibrates t = 0.10 on unspecified example data and fits linear regressions through exactly four scale points; it provides no power analysis, no confidence intervals on the RMSD/MAE values, and no validation on known emergent curves (e.g., from Wei et al. or Schaeffer et al.). With n = 4, a single high-leverage point can dominate the fit, and residual-based statistics have low power to distinguish a sharp unpredictable jump from ordinary scatter around a line. The authors should add a positive-control experiment using synthetic or real emergent curves, and a sensitivity analysis over t and over the set of scale points.","section":"Section IV.A and Section III"},{"comment":"The paper's own results demonstrate that the verdict can flip depending on the residual estimator: for prompt1 on the perturbed bug-fixing task, RMSD exceeds t = 0.10 while MAE does not (0.1037 vs. 0.0890 for BLEU, 0.1145 vs. 0.0952 for CodeBLEU), and the authors explicitly decline to draw a conclusion. This is a concrete example of the classification depending on the hand-set threshold and the chosen formula. The paper should report how results vary across a range of t values, justify the choice of RMSD versus MAE as the primary statistic, or adopt a predefined consistency rule that requires both criteria to exceed the threshold.","section":"Table 2 and Section V.A"},{"comment":"The RQ3 commit message generation results are at the floor: B-Moses is exactly zero at all four scales, B-Norm is near zero (0.0017 RMSD and 0.0015 MAE), and the paper states that bootstrapping was skipped because grades were almost universally zero and that many diffs could not be processed due to model resource limits. A pipeline that reports p = 0 <= t for a model that never performs the task is not discriminating between smooth growth and absence of capability. The RQ3 conclusion should be reframed as 'the models could not execute the task under the tested setup' rather than as evidence of non-emergence.","section":"Section IV.D and Table 4"},{"comment":"The operationalization of emergence as deviation from a linear fit is largely circular. The paper defines emergence as sharp, unpredictable jumps but measures it as residual magnitude from a linear regression chosen by the authors, without justifying that linear growth is the correct null model for these tasks or that residual magnitude is a valid proxy for 'unpredictability.' The Discussion's observation that performance 'roughly increases linearly with model scale' (Section V.D) is derived from the same linear fits used for detection, making the conclusion partly self-confirmatory. The authors should validate the linear null model against alternative functional forms (e.g., log-linear, step functions) or explicitly characterize the non-linear patterns the detector is designed to catch.","section":"Section III and Section VIII"},{"comment":"The threshold t = 0.10 is applied identically to all metrics, but the observed score ranges differ drastically: EM is 0 or near 0, BLEU and CodeBLEU are mostly below 0.3, and B-Moses/B-Norm are near 0. An absolute threshold can be simultaneously too lax for high-variance metrics and too strict for low-variance ones. The paper should calibrate t per metric, normalize residuals (e.g., by the metric's observed range or by the residual standard deviation), and report how the conclusions change under such normalization.","section":"Section III and Section IV.A"}],"minor_comments":[{"comment":"The abstract contains the typo 'do not not provide evidence'; the same phrase appears in the introduction and should be corrected to 'do not provide evidence.'","section":"Abstract and Section I"},{"comment":"The phrase 'largerly underexplored' is a typo for 'largely underexplored.'","section":"Section II.A"},{"comment":"The caption states that prompt2 under CodeBLEU achieves 'RMSD and MAE > 0.40', but the table values are 0.0442 and 0.0401; the caption also says 'few values occur outside the range [0, 0.03)', which is inconsistent with the prompt2 CodeBLEU values being outside that range. The caption should be corrected to reflect the actual numbers.","section":"Table 3 caption"},{"comment":"The sentence 'This variation indicates that the models are not overly biased or simply parroting the expected responses' is an unsupported interpretation of the Exact Match scores, which are zero or near zero; the paper should clarify what the EM values actually show.","section":"Section V.A"},{"comment":"The regression statistic is denoted 'p', which is easily confused with a statistical p-value. Consider renaming it (e.g., 'regression residual measure' or 'deviation statistic') to avoid implying inferential significance, since no confidence intervals or hypothesis tests are computed.","section":"Section III"},{"comment":"The description of the threshold calibration is vague: 'generating example data representing smooth and emergent curves' does not specify the generating process, the number of examples, or the criteria for 'approximately separates.' Provide the full calibration procedure, either in the text or in the replication package, so the threshold is reproducible.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites the first author's unpublished master's thesis [17] as the source of several figures and tables. While this is not inappropriate, the authors should ensure the thesis is accessible and clearly marked as the origin of those results. The paper fits the scope of a software engineering venue and the replication package is a positive step, but the methodological gaps around detector sensitivity and threshold justification need to be addressed before the negative claim can be considered solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is the first systematic attempt to measure emergent capabilities on real software-engineering tasks for an open model family, and second, its central negative finding is plausible but not yet airtight because the detection method is underpowered.\n\nWhat is genuinely new: the authors define emergence operationally as deviation from a linear fit across model scale, apply that to CodeGen1-Multi on bug fixing, code translation, and commit message generation, and find no evidence of sharp, unpredictable performance jumps. They use several prompts per task, both continuous and discontinuous metrics, and they include bootstrapped spread. The paper is honest in its language: the abstract says \"no evidence,\" not \"no emergence,\" and the threats-to-validity section explicitly names prompting dependence and metric limitations. They also ship a replication package.\n\nNow the soft spots, which are real. The detector fits a linear regression through exactly four scale points. That is very little data to distinguish a genuine emergence jump from ordinary scatter or a single high-leverage point. The threshold t=0.10 was calibrated on the authors' own generated example curves, not against known emergent behavior, and there is no power analysis or confidence intervals. The stress-test note is correct that RQ3 is at floor: B-Moses is exactly zero at every scale, so \"non-emergent\" is trivially true and tells you nothing. Also, Table 2 shows one prompt where RMSD exceeds 0.10 but MAE does not; the authors acknowledge this but leave the ambiguity unresolved. These issues do not sink the paper, but they mean the finding is suggestive rather than definitive.\n\nOne more mild complaint: the discussion says the absence \"gives credence to the general consensus that emergent capabilities may be an illusory phenomenon, even in SE,\" which generalizes beyond a single model family. The authors qualify it later, but that sentence overreaches.\n\nWho is this for? Researchers working on LLM evaluation for code, especially those interested in scaling laws or emergence. It would be a reasonable reading-group topic because it raises a methodological question worth debating: what should count as evidence for emergence?\n\nMy recommendation: send it to peer review. The question is important, the study is transparent, and the weaknesses are fixable with a sensitivity analysis or synthetic validation. A serious referee could push for exactly that, and the paper would be stronger for it.","headline":"A careful, honest negative-result study on emergence in SE, but the detector is too weakly validated to fully support the conclusion.","tokens_in":15542,"tokens_out":1982,"would_cite":false,"duration_ms":19767,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Testing CodeGen1-Multi from 350M to 16.1B parameters on bug fixing, code translation, and commit message generation, this paper finds no evidence of emergent capabilities.","keywords":["emergent capabilities","large language models","software engineering","model scaling","bug fixing","code translation","commit message generation","EMERGENTEVAL"],"falsifier":"Run EMERGENTEVAL on CodeGen1-Multi with additional scale points (for example 1B, 3B, 10B, and 20B parameters) on the same three tasks; if any unperturbed task shows a positive performance jump with regression values $p > 0.10$ under both RMSD and MAE, the paper's no-emergence conclusion for that task would be overturned. A second check would be to rerun the perturbed bug-fixing prompt 1 case and see whether the RMSD crossing persists with a larger sample and whether MAE also crosses once the test harness accommodates multi-line output.","tokens_in":14602,"feed_emoji":"🐛","tokens_out":9833,"duration_ms":77686,"temperature":0.7,"pith_summary":"This paper asks whether large language models for software engineering acquire emergent capabilities—abilities that appear suddenly and unpredictably only after a model reaches a certain size—rather than improving smoothly with scale. To answer, the authors build EMERGENTEVAL, a model-agnostic pipeline that grades a model family on a task at several model sizes and measures how far the scores deviate from a linear growth curve. They apply it to the CodeGen1-Multi family at 350M, 2.7B, 6.1B, and 16.1B parameters, covering bug fixing, code translation, and commit message generation. In the unperturbed tasks, all regression values fall below the pre-set threshold $t=0.10$, and the one perturbed case that crosses it represents a performance drop, not a jump; the paper reports no evidence of emergent capabilities. If correct, this means practitioners should not expect unpredictable capability jumps from scaling up these models within the tested range.","feed_headline":"No evidence of emergent coding skills from 350M to 16.1B","feed_subtitle":"A new pipeline tests bug fixing, code translation, and commit messages: performance grows smoothly, not suddenly.","key_machinery":"The load-bearing object is EMERGENTEVAL, a pipeline whose inputs are a task $K$, a model family $F$, one or more metrics $M$, a regression method $R$, and a threshold $t$. The regression fits expected scores from the available model scales, and the deviation between observed and predicted scores is compressed into a single regression value $p$ using RMSD or MAE; if $p > t$, the capability is classified as emergent, otherwise as predictable growth. The case studies instantiate this with linear regression, $t = 0.10$ set before testing, bootstrapping at subsample size 50 over 500 iterations, and perturbation-based checks using CheckList and Levenshtein distance, so that 'unexpected jump' becomes an operational, quantitative deviation from linearity.","core_discovery":"On its own terms, the central result is negative: for the three selected software-engineering tasks and the prompts tested, CodeGen1-Multi does not show emergent capabilities between 350M and 16.1B parameters. Bug fixing and code translation scores under BLEU and CodeBLEU rise only slightly and roughly linearly with scale, Exact Match stays near zero, and commit message generation scores are near zero at every scale. The pipeline's regression values $p$, computed with RMSD and MAE against a linear fit, are almost always far below the threshold $t = 0.10$; the single crossing, on perturbed bug-fixing inputs under prompt 1, comes from a sharp decline at the largest scale and is read as negative rather than positive emergence. The paper concludes that over the observed scales, scaling produces modest, roughly linear gains at best, and that this pattern is consistent with treating apparent emergence as a phenomenon that depends heavily on the metric, the prompt, and the task.","pith_inferences":["With only four scale points and a linear baseline, the detector may be too coarse to catch emergence that appears at higher scale or in a narrow size window; the paper itself does not claim to rule out emergence outside the tested configurations.","The perturbed bug-fixing case shows the verdict can depend on the deviation formula: RMSD crosses the threshold while MAE does not, so a stricter detector would require both metrics to agree before declaring emergence.","If emergence is metric-dependent, as earlier work cited by the paper argues, then this negative result does not automatically transfer to other software-engineering tasks or to model families with denser coverage of the scale axis.","A natural next experiment is to run the same pipeline on a model family with many released checkpoints and additional SE tasks, using semantic-equivalence metrics as a complement to n-gram overlap."],"forward_implications":["For CodeGen1-Multi between 350M and 16.1B parameters, scaling alone does not unlock new task abilities on the three tasks tested; gains are modest and roughly linear.","Prompting strategy appears to affect performance more than model scale, and one prompt made the largest model perform worse, so scale-up decisions should be tied to measured gains rather than assumed emergence.","EMERGENTEVAL gives a quantitative alternative to visual inspection of scaling curves, and it can flag both positive and negative deviations from expected growth.","Within this scale range, compute budgeting for bug fixing, code translation, or commit message generation should not assume an unpredictable capability jump will appear.","The negative results suggest that conclusions about emergence drawn from natural-language tasks may not transfer automatically to software engineering tasks."],"supporting_citations":[{"why":"Defines emergent abilities as sharp, unpredictable performance jumps at sufficient scale, which is the phenomenon this paper operationalizes.","marker":"[1]"},{"why":"Introduces the CodeGen1-Multi model family at the four scales used in the case studies.","marker":"[3]"},{"why":"Argues that apparent emergence can be an artifact of discontinuous metrics, motivating the paper's use of continuous metrics and regression.","marker":"[5]"},{"why":"Supplies CheckList, used to generate the perturbed test cases for robustness checks.","marker":"[8]"},{"why":"Defines CodeBLEU, used to grade code translation and bug-fixing outputs against abstract syntax tree structure.","marker":"[11]"},{"why":"Provides the CodeXGLUE benchmark, the source of the bug-fixing and code-translation test sets.","marker":"[13]"},{"why":"Supplies the CoDiSum dataset used for the commit message generation task.","marker":"[15]"},{"why":"A prior study of commit message generation whose metric choices (B-Moses, B-Norm) are reused here.","marker":"[16]"},{"why":"Describes the evaluation of commit message generation models and the metric choices applied in the third case study.","marker":"[18]"}],"fun_headline_variants":["Scaling LLMs fails to spark emergent software engineering","No emergent coding abilities across LLM scales in SE tasks","Bigger LLMs, same coding: no sudden skill jumps in SE","LLM scaling yields smooth gains, not emergent coding skills","Testing 350M to 16.1B: no emergent capabilities in SE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole negative finding rests on one premise: that how far four measured scores fall from a straight line, compared with a cutoff of 0.10 chosen before the test, reliably detects whether a capability has emerged.","fun_headline_variants_meta":{"raw":{"variants":["Scaling LLMs fails to spark emergent software engineering","No emergent coding abilities across LLM scales in SE tasks","Bigger LLMs, same coding: no sudden skill jumps in SE","LLM scaling yields smooth gains, not emergent coding skills","Testing 350M to 16.1B: no emergent capabilities in SE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1465,"prompt_tokens":1002,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":375}},"tokens_in":618,"tokens_out":463,"duration_ms":4158,"temperature":1.0,"reasoning_tokens":375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:41:48.343796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EMERGENTEVAL on CodeGen1-Multi with additional scale points (for example 1B, 3B, 10B, and 20B parameters) on the same three tasks; if any unperturbed task shows a positive performance jump with regression values $p > 0.10$ under both RMSD and MAE, the paper's no-emergence conclusion for that task would be overturned. A second check would be to rerun the perturbed bug-fixing prompt 1 case and see whether the RMSD crossing persists with a larger sample and whether MAE also crosses once the test harness accommodates multi-line output.","supporting_citations":[{"cited_title":"Emergent abilities of large language models,","cited_arxiv_id":null,"evidence_quote":"Defines emergent abilities as sharp, unpredictable performance jumps at sufficient scale, which is the phenomenon this paper operationalizes."},{"cited_title":"Codegen: An open large language model for code with multi-turn program synthesis,","cited_arxiv_id":null,"evidence_quote":"Introduces the CodeGen1-Multi model family at the four scales used in the case studies."},{"cited_title":"Are emergent abilities of large language models a mirage?","cited_arxiv_id":null,"evidence_quote":"Argues that apparent emergence can be an artifact of discontinuous metrics, motivating the paper's use of continuous metrics and regression."},{"cited_title":"Beyond accuracy: Behavioral testing of nlp models with checklist,","cited_arxiv_id":null,"evidence_quote":"Supplies CheckList, used to generate the perturbed test cases for robustness checks."},{"cited_title":"Codebleu: a method for automatic evaluation of code synthesis,","cited_arxiv_id":null,"evidence_quote":"Defines CodeBLEU, used to grade code translation and bug-fixing outputs against abstract syntax tree structure."},{"cited_title":"Commit message generation for source code changes,","cited_arxiv_id":null,"evidence_quote":"Supplies the CoDiSum dataset used for the commit message generation task."},{"cited_title":"Using large language models for commit message generation: A preliminary study,","cited_arxiv_id":null,"evidence_quote":"A prior study of commit message generation whose metric choices (B-Moses, B-Norm) are reused here."},{"cited_title":"On the evaluation of commit message generation models: An experimental study,","cited_arxiv_id":null,"evidence_quote":"Describes the evaluation of commit message generation models and the metric choices applied in the third case study."}],"review_version":1}