{"id":"187330b3-3392-4cb6-bad7-7ab55544894d","arxiv_id":"2506.06508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using Shannon entropy over source-code tokens and AST edges, the paper finds weak correlations with classic complexity metrics and achieves 37-83% precision in detecting unusual commits across 95 Java projects.","lead":"This paper measures how the 'entropy' of source code, based on token and syntax-tree frequencies, changes across the full commit history of 95 open-source Java projects. It reports that entropy-based metrics correlate only weakly with classic complexity metrics and can flag unusual code changes with measured precision between 37% and 83%, depending on settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision claims rest on author-generated labels, not maintainer ground truth; Section 7 admits maintainer-validated precision could be 'vastly inferior.' A maintainer relabeling study is needed before the >60% claim is credible.","rationale":"I agree with the reader's weakest-assumption: the precision numbers are the paper's central quantitative contribution, and they are anchored to labels produced by people who are explicitly not domain experts on the projects. This is not a minor limitation: the definition of 'unusual/needs prioritization' is inherently a maintainer judgment, yet the authors substitute their own judgment after reading commit pages and doing brief research. The paper is admirably candid about this in Section 7, and even proposes the maintainer study as future work. That candor is credit-worthy, but it also confirms that the claimed >60% precision is not yet established for the intended use case. Additional secondary concerns—the post-hoc selection of the z-score threshold based on 'best signal-to-noise ratio,' the lack of baseline comparisons, and the abstract's 'over 60%' vs. the 58.12% most-conservative estimate—are real but do not displace the ground-truth issue. If maintainer labels disagree substantially, no amount of threshold tuning or baseline work can rescue the precision claim. The proposed test is the definitive check, and until it is run, CONDITIONAL (or UNCHANGED from the reader's CONDITIONAL verdict) is the right call.","tokens_in":29996,"tokens_out":5459,"duration_ms":54004,"concrete_test":"Conduct a maintainer ground-truth study: randomly sample 100–150 of the 974 manually analysed events across 20–30 of the 95 projects. For each event, recruit a current maintainer or core contributor of that project, blind to the authors' labels, and ask them to judge, using the same procedure and definition, whether the change 'would need to be prioritised by maintainers' (and optionally to categorize it). Compute precision separately for the three strictness levels using the maintainer labels. If maintainer-validated precision under the most conservative definition is below 60%, the abstract claim is not supported; if it is at or above 60%, the concern is resolved in favor of the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'may effectively recognise unusual source code change events with over 60% precision'—is supported only by precision estimates (Table 7) computed against labels assigned by the authors, not by the maintainers of the 95 repositories. The authors identify this as the 'foremost internal threat' (Section 7): 'we are not involved in the development of the repositories we analysed, we have no domain knowledge... the precision of our detection may be vastly inferior to that obtained if events were labelled by the actual maintainers.' The label definition is inherently subjective: a true positive is a change 'that would need to be prioritised by maintainers.' Inter-rater agreement was only α=0.61 before discussion (Section 4.4.1), and after discussion the raters may have converged on a shared subjective standard, not on the maintainers' standard. Precision also varies from 58.12% (most conservative) to 83.13% (least conservative) depending on which arbitrary strictness level is selected; the abstract's 'over 60%' is actually below 60% in the most conservative row. Because the entire RQ4 evaluation rests on these labels, the quantitative claim is unverified until ground truth comes from the projects' own stakeholders.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines Shannon-entropy metrics for two source-code representations (token streams and abstract syntax tree edges), mines the complete commit histories of 95 actively maintained Java GitHub projects, and investigates four research questions: trends of entropy evolution (RQ1), correlations with classic complexity metrics (RQ2), agreement with Lehman's laws of software evolution (RQ3), and entropy-based anomaly detection of unusual source-code changes (RQ4). The central quantitative claim is that the proposed detection approach can recognize unusual change events with over 60% precision, based on manually labeled samples of outlier commits.","tokens_in":30217,"tokens_out":2967,"duration_ms":32048,"significance":"If validated, the paper would provide a lightweight, deterministic, and language-agnostic signal for identifying unusual source-code changes, complementing existing complexity metrics and offering a reproducible basis for future information-theoretic work on software evolution. The study's strengths include its scale (95 projects, 1,827,204 change events), commit-by-commit granularity, promised availability of code and data, explicit reporting of inter-rater agreement, and unusually candid treatment of threats to validity. However, the headline precision claim is currently supported only by labels assigned by the authors themselves, and the detection threshold appears to be selected on the same labeled data, so the practical significance of the >60% figure remains conditional on external validation.","major_comments":[{"comment":"The central claim that the approach 'may effectively recognise unusual source code change events with over 60% precision' is measured against labels assigned by the authors, not by the maintainers of the 95 projects. Section 7 explicitly concedes that 'the precision of our detection may be vastly inferior to that obtained if events were labelled by the actual maintainers.' Moreover, Table 7 shows that the most conservative strictness level yields 58.12% precision for the 100% memory configuration, so the abstract's 'over 60%' is not supported by the most conservative row. The paper should either obtain maintainer ground-truth labels for a validation sample or reframe the claim as a proof-of-concept precision estimate under author-defined relevance, with the associated uncertainty made explicit.","section":"Abstract; §4.4.1; Table 7"},{"comment":"The z-score threshold used to define outliers appears to be tuned on the same manually labeled data used for evaluation. Section 4.4.1 states that any change with z-score less than or equal to 3 was considered a false positive, while Section 7 reveals that the authors selected 'the definition of statistical outlier (i.e., z-score) for which we observed the best signal-to-noise ratio.' This in-sample threshold selection, combined with the additional free parameters of maximum memory size and the 50% feature-code contribution filter, means the precision estimates in Table 7 are likely optimistic. The authors should report how the threshold was chosen, justify it independently of the evaluation labels, or provide a sensitivity analysis that separates threshold selection from precision measurement.","section":"§4.4.1; §7"},{"comment":"The precision results are highly sensitive to the maximum memory configuration, yet the manuscript presents only the 100% memory setting as supporting the abstract's claim. Table 7 reports precision drops to 40.12% at 50% memory even in the least conservative strictness level, and the RQ4 summary notes that reducing memory increases detected events by about 20% but decreases precision. Since the choice of maximum memory is a free parameter with no clear a-priori justification, the paper should either justify the full-memory configuration as the intended deployment setting or present the precision claim across configurations with a clear recommendation.","section":"§4.4.1; Table 7"}],"minor_comments":[{"comment":"Equation (2) is missing the minus sign that appears in Equation (1); as written, H(X) would be negative for a non-degenerate distribution.","section":"§1.2, Eq. (2)"},{"comment":"There are several typos: 'cyclomacic complexity' should be 'cyclomatic complexity', 'Halmstead' should be 'Halstead', and 'README.me' should be 'README.md'.","section":"§1.1, §5, §10"},{"comment":"The table caption reads 'increasing percentual changes of information content' but the table reports counts of spikes and drops; please clarify whether these are absolute percentage changes and how the spike/drop rows are defined.","section":"Table 3"},{"comment":"The text and Figure 10 refer to a sample of 350 points for the full context window, while Table 6 lists a 95% confidence-interval sample of 320 points and the accompanying text says n = 321; these numbers should be reconciled.","section":"§4.4.2, Figure 10, Table 6"},{"comment":"The comparison with ChatGPT-4 is largely speculative and not grounded in the empirical data collected for the other research questions; consider moving it to a clearly marked discussion subsection or shortening it so it does not distract from the paper's contributions.","section":"§6"},{"comment":"The paper refers to 'structural entropy obtained from the edges of the respective AST' but does not explain whether edge types, node types, or both are counted; please clarify the exact histogram used for Equation (4).","section":"§3.2, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is already formatted as an accepted paper for Empirical Software Engineering, which suggests the review context may be unusual; nevertheless, my assessment concerns the scientific content as presented. The core weakness is that the paper's headline quantitative claim rests on author-generated labels and in-sample threshold tuning, both of which are acknowledged in Section 7. This is fixable with a maintainer-validation study or a careful reframing of the claim, so I do not recommend rejection. The speculative sections (Surprisal Index, ChatGPT comparison) could be trimmed to improve focus without affecting the empirical contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, large-scale empirical study of Shannon entropy applied to token streams and AST edges across the full commit histories of 95 Java projects. The entropy definitions themselves are not new—the textual one is Berlinger/Cook, and the structural one is a direct analogue on AST edges—but the scale (1.8M change events, 294k commits) and the anomaly-detection evaluation are. If you work on software metrics or repository mining, this is worth reading; the RQ4 precision claim, though, should be treated as conditional.\n\nWhat the paper does well: the commit-by-commit traversal is a real step beyond the usual 100- or 500-commit slicing; the correlation analysis with classic complexity metrics is honest, with the authors openly reporting weak correlations and drawing only modest conclusions; and the Lehman-laws analysis, especially the disagreement with the smooth-growth laws, is a legitimate empirical observation. The threats-to-validity section is unusually candid: it admits the labels are not maintainer ground truth and that maintainer-validated precision could be vastly inferior.\n\nThe soft spots are the ones flagged in the stress test, and they are real. The >60% precision figure in the abstract is computed against labels assigned by the authors, not by the maintainers of the 95 repositories. Inter-rater agreement was only alpha=0.61 before discussion and 0.87 after, so the raters may have converged on their own shared standard rather than the maintainers'. The z-score threshold was selected after observing signal-to-noise on the same labelled data, and no baseline detector (e.g., simple lines-changed outliers or random commits) is compared. The per-file entropy inflation acknowledged in Section 7 could also skew which events are flagged. In Table 7, the most conservative row under 100% memory is 58.12%, so the abstract's 'over 60%' is slightly overstated for that reading. These issues do not invalidate RQ1–RQ3, but they make RQ4 a proof-of-concept rather than a validated precision claim.\n\nMinor points: Section 5's Surprisal Index and GUI mockups are clearly labelled speculative and add little, and Section 6's ChatGPT comparison is high-level and not benchmarked. Neither is fatal.\n\nWho this is for: software engineering researchers interested in code metrics, evolution, and anomaly detection. Practitioners should wait for maintainer validation. Recommendation: send it to peer review—it deserves a serious referee. The revision should either add a maintainer-labeling study or soften the precision claim and add a baseline comparison. I would probably not cite the precision number in my own work, but I would cite the large-scale entropy evolution results.","headline":"Large-scale entropy-evolution study with a candid but author-labelled precision claim; worth peer review, but the >60% anomaly-detection number needs maintainer labels and a baseline before it becomes credible.","tokens_in":30745,"tokens_out":1959,"would_cite":true,"duration_ms":20828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shannon entropy of source code flags unusual commits with over 60% precision.","keywords":["information theory","entropy","source code analysis","software evolution","anomaly detection","abstract syntax tree","software complexity","Lehman's laws"],"falsifier":"Ask the maintainers of the sampled projects to label the detected outlier commits as surprising or routine without seeing the researchers' labels, then recompute precision against those maintainer labels; if agreement is no better than chance or precision falls below 60%, the paper's central claim fails.","tokens_in":29773,"feed_emoji":"📈","tokens_out":11842,"duration_ms":108331,"temperature":0.7,"pith_summary":"This paper tries to establish that the entropy of source code, measured over token frequencies and over abstract-syntax-tree edge types, provides a practical signal for how surprising a commit is. Tracking every commit of 95 actively maintained Java projects, the authors find that commits whose entropy delta is an outlier against recent history are judged relevant by human review with over 60% precision, and up to 83% under a permissive definition. They also report that these entropy metrics correlate only weakly with McCabe's cyclomatic complexity and line/token counts, which they interpret as evidence that entropy captures a different dimension of code complexity. If the paper is right, a cheap per-commit entropy calculation could help maintainers decide where to focus their attention without running heavyweight analyses.","feed_headline":"Entropy flags unusual code changes with over 60% precision","feed_subtitle":"A per-commit entropy delta from tokens and AST edges can surface the changes worth a maintainer's attention.","key_machinery":"The mechanism is the per-commit entropy delta. Textual entropy is the Shannon entropy of the token histogram, $H_{\\text{TOKEN}} = -\\sum_{w\\in W} p(w)\\log_2 p(w)$, and structural entropy is the same calculation over the frequencies of abstract-syntax-tree edge types. After each commit, the total entropy of the changed files is recomputed and compared with the previous state, producing a delta that is added to a running distribution of past deltas; a commit is flagged as unusual when its delta has a z-score of at least 3. A configurable memory window determines how many past deltas remain in the distribution, and the study varies this window at 50%, 75%, and 100% of the commit history.","core_discovery":"The paper's central claim is that the Shannon entropy of a project's source code, measured over token frequencies and over abstract-syntax-tree edge types, captures how much information each commit inserts or removes, and that large entropy deltas mark changes a maintainer would want to examine. Textual entropy is $H_{\\text{TOKEN}} = -\\sum_{w \\in W} p(w)\\log_2 p(w)$, and structural entropy applies the same formula to the histogram of AST edge types. Over 1,827,204 change events from 95 Java projects, commits whose delta against the recent distribution of deltas exceeded a z-score threshold were judged by the authors' manual review to be relevant in 58–83% of cases, with 72.5% under their main conservative definition. The same data yield weak correlations with McCabe's cyclomatic complexity and with line/token counts, leading the paper to claim that entropy measures a different dimension of complexity, and support Lehman's laws of continuing change and increasing complexity while contradicting the smooth-growth and organizational-stability laws.","pith_inferences":["The paper does not test it, but the memory-size results suggest the z-score threshold should be adapted to the local slope of entropy growth, so that early steep phases and later stable phases receive different outlier criteria.","The paper argues for language agnosticism from representation histograms, yet only studies Java; a direct check is to run the same detector on Python or JavaScript repositories and see whether maintainer-validated precision holds above 60%.","Because entropy correlates weakly with cyclomatic complexity, a fused surprisal index may catch changes neither metric flags alone; the correlation tables in the paper make that a concrete next experiment rather than a mere suggestion."],"forward_implications":["On the full-history setting, flagged commits were judged relevant in 58% of cases under the most conservative definition and 83% under the least conservative, with 72.5% at the paper's main conservative level.","Entropy metrics correlate weakly with McCabe cyclomatic complexity (range -0.05 to 0.32) and with line/token counts, indicating that information-theoretic measures capture a different dimension of code complexity.","About two-thirds of detected unusual events involve internal, programmer-facing concerns, while user-facing changes are rare (about 10%) but tend to touch many files at once.","Limiting the memory window from full history to 50% of commits raises the number of detected events by roughly 20% but lowers precision, meaning detection thresholds must track the project's current pace of entropy change.","Lehman's laws of continuing change and increasing complexity are supported by the entropy data, while the smooth-growth and organizational-stability laws are not in this open-source sample."],"supporting_citations":[{"why":"Supplies the definition of entropy used to build the textual and structural metrics.","marker":"(Shannon, 1948)"},{"why":"States the laws of software evolution that the study tests against entropy time series.","marker":"(Lehman, 1980)"},{"why":"Defines the cyclomatic complexity baseline that entropy metrics are correlated with.","marker":"(McCabe, 1976)"},{"why":"Establishes change entropy over commit metadata, which this paper contrasts with measuring the source code itself.","marker":"(Hassan, 2009)"},{"why":"Provides the practitioner precision expectations used to judge whether entropy-based detection is practical.","marker":"(Christakis and Bird, 2016)"},{"why":"Motivates unusual-event detection for maintainers and supplies the labeling procedure this study adapts.","marker":"(Treude et al., 2018)"},{"why":"Provides the repository-mining framework used to traverse entire commit histories.","marker":"(Spadini et al., 2018)"},{"why":"Supports the naturalness-of-software premise underlying language-agnostic token entropy.","marker":"(Hindle et al., 2016)"}],"fun_headline_variants":["Entropy deltas flag unusual code changes at 72.5% precision","Code entropy reveals a different axis of complexity than classic metrics","Per-commit entropy spots changes worth a maintainer's review","Information theory measures code evolution beyond complexity metrics","Textual and structural entropy detect anomalous commits with 72.5% precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the research team's judgement of which outlier commits are important matches what the projects' own maintainers would consider important.","fun_headline_variants_meta":{"raw":{"variants":["Entropy deltas flag unusual code changes at 72.5% precision","Code entropy reveals a different axis of complexity than classic metrics","Per-commit entropy spots changes worth a maintainer's review","Information theory measures code evolution beyond complexity metrics","Textual and structural entropy detect anomalous commits with 72.5% precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2555,"prompt_tokens":953,"completion_tokens":1602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1515}},"tokens_in":569,"tokens_out":1602,"duration_ms":10076,"temperature":1.0,"reasoning_tokens":1515,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:56:28.854042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask the maintainers of the sampled projects to label the detected outlier commits as surprising or routine without seeing the researchers' labels, then recompute precision against those maintainer labels; if agreement is no better than chance or precision falls below 60%, the paper's central claim fails.","supporting_citations":[{"cited_title":"In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 332--343","cited_arxiv_id":null,"evidence_quote":"Provides the practitioner precision expectations used to judge whether entropy-based detection is practical."},{"cited_title":"Journal of Systems and Software 142:237--247, doi:https://doi.org/10.1016/j.jss.2018.04.063, ://www.sciencedirect.com/science/article/pii/S0164121218300876","cited_arxiv_id":null,"evidence_quote":"Motivates unusual-event detection for maintainers and supplies the labeling procedure this study adapts."},{"cited_title":"Communications of the ACM 59(5):122--131","cited_arxiv_id":null,"evidence_quote":"Supports the naturalness-of-software premise underlying language-agnostic token entropy."}],"review_version":1}