{"id":"ad33980b-e22a-4bcb-85cf-c349c72f1a36","arxiv_id":"2608.03007","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AI-generated commits and squash-merging violate the human-origin assumption underlying software measurement, and the field should run a large-scale AI-assisted replication agenda.","lead":"A reflection paper argues that a core assumption in software measurement, that version-control traces come from human developers, is breaking as AI agents and squash-merge workflows reshape the data. It adds new statistics on squash-merging prevalence and calls for an AI-assisted replication program to re-test classic findings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'frequently violate' claim rests on an unquantified AI-agent commit share; without measurement the central urgency lacks support.","rationale":"The reader's weakest assumption identified the same load-bearing premise: AI-agent trace share is asserted, not quantified. I agree that this is the most important gap. The paper's own squash-merging analysis (Section 3.1) demonstrates the authors know how to quantify a practice's prevalence, which makes the absence of a similar estimate for AI-agent commits especially conspicuous. The proposed test would settle whether the premise holds: if AI agents already account for a large fraction of commits, the central claim is supported; if not, the paper becomes a forward-looking caution rather than a description of current irrelevance. This does not change the reader's CONDITIONAL verdict; it reinforces it, so the verdict remains UNCHANGED.","tokens_in":14322,"tokens_out":2603,"duration_ms":27378,"concrete_test":"Using the World of Code archive or GH Archive for 2024-2026, sample all public projects with more than 1,000 commits and apply a validated bot/agent classifier (e.g., Dey et al. MSR 2020 heuristics plus LLM-based author attribution on a random subset) to estimate the fraction of commits and pull requests attributable to AI agents, stratified by year and project size. Compare against a 2018-2020 baseline. If the AI-agent commit share in the most recent year is below 5%, or if the classifier cannot reliably distinguish hybrid human+AI commits, the 'frequently violate' premise lacks empirical support and the paper's urgency should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central argument (Abstract, Section 3.2) is that AI agents now produce a substantial share of version-control traces, making the human-origin assumption 'frequently' violated and thereby rendering current trace-based measurement of human behavior irrelevant. The only evidence cited is adoption studies [69, 82, 86] that show developers and organizations adopting AI agents, not the share of commits or pull requests actually authored by agents in the traces researchers mine. The World of Code-based analysis in Section 3.1 quantifies squash-merging prevalence (39% of projects), but no analogous quantification is provided for AI-agent trace share. The word 'frequently' is doing enormous work: if AI-authored commits are, say, under 1% of commits in public repositories, then the foundational assumption is only occasionally violated, and the claim that 'the current way of software measurement via trace data would no longer be relevant' is hyperbolic. Moreover, the paper itself notes that human commits are now 'hybrid artifacts' when Copilot is used, which makes the binary human/AI distinction fuzzy and leaves the threshold for 'violation' undefined. Without a quantitative estimate, the call for a systematic replication program is a reasonable precaution but not a demonstrated necessity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This reflection paper argues that empirical software engineering's reliance on tool traces (version control, issue trackers) rests on assumptions that are increasingly violated by modern development practices: commits as atomic units, human origin of traces, append-only history, and stable author identity. The authors provide new descriptive evidence on squash-merging: in eight curated GitHub projects, 35–65% of merged pull requests contain more than one commit (Table 2), and a World of Code analysis estimates that 39% of large public projects contain at least one squash-merged commit, with adoption rising from 18% in 2016 to 39% in 2025. They argue that AI agents now generate traces that 'frequently' violate the human-origin assumption, rendering current measurement methods irrelevant, and they propose a systematic AI-assisted replication program, new measurement constructs, and provenance metadata support from tool makers.","tokens_in":14492,"tokens_out":5028,"duration_ms":47776,"significance":"If substantiated, the paper identifies a genuinely important threat to the validity of a large body of software-engineering research and proposes a concrete response. The squash-merging statistics are a useful contribution, and the availability of scripts (Zenodo [11]) supports reproducibility. The paper is appropriately cautious in several places, acknowledging the detection heuristic's undercount and the need for human validation of AI-assisted replications. However, the paper's central urgency claim—that AI-generated traces are already prevalent enough to make current measurement irrelevant—is not backed by a quantitative estimate of AI-authored trace share, and the paper oscillates between binary and continuous conceptualizations of authorship. These gaps do not invalidate the thesis, but they require revision before the call for a replication program can be considered evidence-based.","major_comments":[{"comment":"The paper's central urgency claim—that traces 'frequently violate' the human-origin assumption and that without change 'the current way of software measurement via trace data would no longer be relevant'—rests on the estimated prevalence of AI-generated commits. The only support offered is citations to adoption studies [69, 82, 86], none of which quantifies the share of commits or pull requests in mined repositories that are authored by AI agents. The paper itself notes that human commits are 'hybrid artifacts' (Section 3.2), which makes the binary violation threshold undefined. Please provide a quantitative estimate (even a range) of the proportion of AI-authored or AI-assisted commits in datasets such as World of Code or GitHub Archive, or justify why such a measurement is currently impossible; otherwise the call for a systematic replication program is a reasonable precaution but not a demonstrated necessity.","section":"Section 3.2 and Abstract"},{"comment":"The 39% project-level adoption rate counts a project as adopting squash-merging if it contains at least one commit whose title matches the pattern '(# PR number)'. This conflates occasional use with project-level adoption and is sensitive to the >1000-commit threshold; a project with one squashed PR among thousands of ordinary commits counts identically to a project with mandatory squash-merging. Moreover, the 8-project convenience sample in Table 2 shows that 35–65% of PRs contain more than one commit, but the paper does not report what fraction of commits in the population are lost or how the distribution of squash-merged commit share per project looks. Please report the distribution of squash-merged commits per project (e.g., percentage of merge commits that are squashed) and analyze the sensitivity of the adoption estimate to the detection heuristic and the project-size threshold, so that the claim that traces 'frequently' deviate from atomic-commit semantics is supported.","section":"Section 3.1, Table 2, World of Code analysis"},{"comment":"The 'human origin' assumption is presented as binary while the paper acknowledges a spectrum (human- vs AI-assisted vs AI-generated). Since the violation is defined as 'human origin,' it is unclear what fraction of AI involvement constitutes a violation, and the proposed provenance metadata schema ('Was this commit human-authored, AI-generated, or AI-assisted?') does not resolve this. Please define an operational threshold or a graded measure of human authorship, and state how the proposed replication studies would handle the continuous nature of the construct.","section":"Section 3.2 and Table 1"}],"minor_comments":[{"comment":"The table as typeset lacks clear column separators; for example, 'denoland/deno15 142' reads as part of the project name. Please format the table with explicit column spacing or a ruled table so that the number of PRs is distinguishable from the project name.","section":"Table 2"},{"comment":"The 18% in 2016 and 39% in 2025 project-level adoption numbers are not accompanied by a figure or table in the manuscript; the reader cannot verify this trend. If a plot is available on Zenodo, reference it directly in the text, or add a small figure showing the adoption curve.","section":"Section 3.1"},{"comment":"Figure 2 is referenced but not visible in the provided manuscript text; please ensure the figure file is included and that the caption clearly identifies which commit messages demonstrate the difference between a traditional merge and a squash merge.","section":"Figure 2"},{"comment":"The column header 'Productivity & effort estimation' uses an ampersand; use 'and' for formal text to avoid confusion with the LaTeX alignment character.","section":"Table 3"},{"comment":"The sentence 'Without a significant change, the current way of software measurement via trace data would no longer be relevant' appears verbatim in both the abstract and Section 3.2; consider rewording one of them to avoid exact duplication.","section":"Section 3.2 and Abstract"},{"comment":"The sentence 'This democratized empirical software engineering: Any project using these systems could analyze its own data and benchmark against historical states' would read more clearly with a semicolon or a period after 'software engineering'.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position/reflection paper with one novel empirical data point (squash-merging prevalence). The main weakness is the unquantified AI-agent contribution share, which the authors should either measure or clearly reframe as a risk rather than an established fact. The manuscript fits ASE's scope as a reflection, but the emphasis on 'immediate action' may be perceived as overstatement without supporting data. The squash-merging analysis is a valuable contribution and should be preserved in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this reflection paper does something useful—it names the human-origin assumption explicitly, shows with fresh data that squash-merging is now a mainstream practice, and argues for a systematic replication program. The squash-merging statistics are the genuine new contribution: 39% of large GitHub projects contain at least one squash-merged commit, with adoption rising from 18% to 39% over 2016–2025. The per-project numbers in Table 2 are illustrative but consistent with that. The authors also ship the collection scripts on Zenodo, which is reproducible and welcome.\n\nThe soft spot is exactly where the stress-test note points. The claim that traces 'frequently violate' the human-origin assumption because of AI agents is asserted, not measured. The paper cites adoption studies [69, 82, 86], but none quantify the share of commits actually authored by agents in the repositories researchers mine. If agent commits are still a small fraction, 'frequently' is doing too much work. The authors themselves note that human commits are now hybrid artifacts when Copilot is involved, which makes the binary human/AI distinction fuzzy and raises the question of what counts as a 'violation.' So the central urgency is plausible but not demonstrated. The call for a replication program is reasonable as a precaution; it just isn't forced by the evidence in this paper.\n\nOther soft spots: the 8-project sample is a convenience sample, and the population estimate relies on a detection heuristic the authors admit undercounts. Neither is fatal for a reflection paper, but the numbers should be read as lower bounds.\n\nWhat's honestly good: the framing of foundational assumptions in Table 1 is clean, the connection between changing data-generation practices and the validity of classic findings is well argued, and the proposal for provenance metadata as a first-class commit record is concrete and actionable. Citation practice looks fair; the authors cite prior work on squash merging, bots, and altered histories rather than claiming novelty where none exists.\n\nWho should read this: anyone doing repository mining or building metrics on version-control traces. It's a useful agenda-setting piece, not a definitive empirical study. I'd send it to peer review; it deserves referee time, and the authors have done the field a service by naming the problem and putting a stake in the ground. I'd push the reviewers to ask for either a tempering of the AI-agent claim or a quantification of agent trace share.","headline":"A sincere, well-grounded call to action on the erosion of the human-origin assumption in software measurement, but the 'frequently violate' claim outruns the evidence.","tokens_in":15051,"tokens_out":1807,"would_cite":true,"duration_ms":18135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Software measurement's founding assumption\\u2014that version-control traces are generated by human developers\\u2014is now frequently violated by AI agents, so the field must detect violations and run a systematic AI-assisted replication…","keywords":["software measurement","mining software repositories","human-origin assumption","squash-merging","AI-generated code","replication","provenance metadata","empirical software engineering"],"falsifier":"Measure the actual share of commits, pull requests, and reviews attributable to AI agents in a large public archive such as World of Code, using markers like bot accounts, self-admitted AI usage, and commit-message patterns; if the share is a tiny fraction of all traces, the claim that the human-origin assumption is 'frequently' violated is false. A second check would be to run replications of the three constructs in Table 3 on contemporary data: if commit-count productivity, code-churn defect prediction, and expertise-from-contributions findings all survive unchanged, the paper's central concern is not supported.","tokens_in":14090,"feed_emoji":"🤖","tokens_out":6492,"duration_ms":61695,"temperature":0.7,"pith_summary":"This paper argues that the dominant way of measuring software\\u2014mining traces left in version-control and issue-tracking tools\\u2014rests on an assumption that is silently dissolving: that the traces are generated by human developers building genuine systems. Changes in practice, especially squash-merging pull requests into single commits, and the arrival of AI agents that commit, review, and open pull requests mean contemporary traces no longer reliably reflect human activity. The paper contends that if this continues, classic findings about productivity, defects, and collaboration may not replicate on current data, and software measurement would lose relevance. It calls for detecting assumption violations, developing methods valid under new data-generation regimes, and a systematic AI-assisted replication program that re-runs key findings on modern data. If the paper is right, much of what empirical software engineering believes about how software is built would need to be re-examined.","feed_headline":"AI commits break a core assumption of software measurement","feed_subtitle":"Version-control traces were assumed human; classic findings now need re-testing on today's data.","key_machinery":"The carrying mechanism is the set of foundational assumptions listed in Table 1, with the human-origin assumption as the load-bearing one: a commit is an atomic unit of work, traces are generated by human developers, history is append-only, and author identity is unique and stable. Each assumption is paired with a modern violation\\u2014squash-merging, AI-generated commits, force-pushing and rebasing, fragmented or shared identities\\u2014and the paper uses these pairings to organize why analyses built on the assumptions fail. The worked example of squash-merging does the argumentative work of showing that a purely workflow-driven change can systematically distort commit-count productivity, churn, bug-introduction, collaboration-network, and temporal analyses; the AI-agent example extends the same logic to traces whose generative process is no longer human at all. These examples convert the abstract worry about shifting data generation into concrete, checkable threats to specific measurement constructs.","core_discovery":"The central claim is that the field's foundational 'human-origin assumption'\\u2014that tool traces are produced by professional developers following established processes with quality gates\\u2014is now frequently violated, and that the validity of software-measurement research therefore cannot be taken for granted. The paper supports this with two worked examples: squash-merging collapses multi-commit pull requests into a single commit, erasing development trajectory (with descriptive statistics showing 35\\u201365% of merged PRs contain more than one commit in eight sampled projects and 39% of large active GitHub projects contain at least one squash-merged commit), and AI agents generate commits, messages, and reviews that look human but do not reflect human effort or cognition. From these examples, the paper concludes that current trace-based measurement would no longer be relevant without significant methodological change. The proposed remedy is a systematic, AI-assisted replication program that revisits classic findings on contemporary data, plus advocacy for provenance metadata so future commits can be labeled as human, AI-assisted, or AI-generated.","pith_inferences":["The paper's own proposal contains a feedback loop it does not discuss: if AI assistants carry out the replications, and AI-generated traces are exactly what is undermining the old assumptions, the replications could inherit the same problems; the paper acknowledges in Section 3.3 that AI may misinterpret details and introduce biases, which means the remedy's reliability is not established.","A testable extension would be to estimate the current share of AI-generated commits in public archives and plot its growth; the paper's urgency scales with that share, which it does not measure.","The same assumption-violation logic applies to other trace sources the paper touches only lightly, such as issue trackers and code reviews, where AI agents now file issues and write review comments; a replication agenda could systematically check those traces too.","If squash-merging adoption continues to rise, historical comparisons across time become difficult not only because of AI but because the unit of observation itself changes; one could benchmark new metric definitions against fine-grained branch histories before squashing to quantify the distortion."],"forward_implications":["Commit-count and churn-based productivity metrics become unreliable because squash-merging collapses many commits into one and AI agents can generate many commits quickly.","Bug-introduction analyses such as SZZ cannot localize defects inside squashed commits, because a bug and its fix may be subsumed in the same atomic change.","Classic findings on developer expertise, ownership, and collaboration may not replicate when expertise shifts to prompting and review or when agent commits dilute human authorship.","The field needs new measurement constructs that separate human effort from AI-assisted generation, and methods that remain valid when the human-origin assumption is relaxed.","Tool and platform owners should emit provenance metadata (author type, model, prompt, review decision) as first-class parts of the commit record."],"supporting_citations":[{"why":"Shows how pull-request merge practices affect defect-tracing algorithms, grounding the claim that squash-merging breaks bug-introduction analysis.","marker":"[9]"},{"why":"Data-release companion for the squash-merging descriptive statistics, making the Section 3.1 numbers checkable.","marker":"[11]"},{"why":"Documents already-known violations of foundational assumptions in version-control data, supporting the paper's premise that assumptions can silently break.","marker":"[40]"},{"why":"Provides the repository-scale data (World of Code) used to estimate that 39% of large active GitHub projects contain at least one squash-merged commit.","marker":"[51]"},{"why":"Evidence that altered version histories through force-pushing and rebasing are common, one of the Table 1 assumption violations.","marker":"[68]"},{"why":"Adoption study cited for the claim that coding agents now generate traces via the same tools as humans.","marker":"[69]"},{"why":"Cites it for the shift in empirical software engineering as generative AI changes what traces mean.","marker":"[82]"},{"why":"Empirical study of agentic coding pull requests, cited to support that software produced entirely via prompts is now observable in traces.","marker":"[86]"}],"fun_headline_variants":["AI agents bust software measurement's human assumption","Software measurement's human assumption crumbles under AI","AI commits and squash merges shake software measurement","When traces aren't human, software measurement is skewed","Replicating software measurement in the age of AI agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's urgency rests on the premise, asserted in Section 3.2 and not quantified, that a substantial and growing share of version-control traces is now produced by AI agents, so that the human-origin assumption is 'frequently' violated.","fun_headline_variants_meta":{"raw":{"variants":["AI agents bust software measurement's human assumption","Software measurement's human assumption crumbles under AI","AI commits and squash merges shake software measurement","When traces aren't human, software measurement is skewed","Replicating software measurement in the age of AI agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1226,"prompt_tokens":942,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":558,"tokens_out":284,"duration_ms":3498,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:15:00.153935+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual share of commits, pull requests, and reviews attributable to AI agents in a large public archive such as World of Code, using markers like bot accounts, self-admitted AI usage, and commit-message patterns; if the share is a tiny fraction of all traces, the claim that the human-origin assumption is 'frequently' violated is false. A second check would be to run replications of the three constructs in Table 3 on contemporary data: if commit-count productivity, code-churn defect prediction, and expertise-from-contributions findings all survive unchanged, the paper's central concern is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Data-release companion for the squash-merging descriptive statistics, making the Section 3.1 numbers checkable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents already-known violations of foundational assumptions in version-control data, supporting the paper's premise that assumptions can silently break."},{"cited_title":"Rapaport, L","cited_arxiv_id":null,"evidence_quote":"Evidence that altered version histories through force-pushing and rebasing are common, one of the Table 1 assumption violations."},{"cited_title":"Treude and M.-A","cited_arxiv_id":null,"evidence_quote":"Cites it for the shift in empirical software engineering as generative AI changes what traces mean."}],"review_version":1}