{"id":"b6f44a2f-617c-4951-ba0f-c55361e74f40","arxiv_id":"2412.17298","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An empirical study of 1,262 prompt changes across 243 GitHub repositories shows that developers mainly add and modify prompt components during feature development, rarely document the changes, and sometimes introduce inconsistencies or unintended LLM behavior.","lead":"This paper analyzes 1,262 prompt changes in 243 open-source GitHub repositories to see how developers edit the text prompts that control LLM-powered software. It finds that prompts mostly grow through additions and tweaks during feature work, that few such changes are documented in commit messages, and that edits can cause inconsistent or unexpected model behavior.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RQ4/RQ5 maintenance-activity split rests on an LLM commit classifier whose per-class accuracy on this study's commits is unreported; if feature/fix labels are systematically confounded, the 'most changes during feature development' claim (59.7%) may not hold.","rationale":"The paper is a methodical empirical study with genuine manual coding, inter-rater reliability reporting (Jaccard 65%, Section IV-D), and concrete examples for the inconsistency findings, which are independent evidence for the 'additions and modifications dominate' and 'inconsistencies occur' observations. The most fragile link in the central quantitative claim is the LLM-based commit classification: the 59.7% feature-development figure and the RQ5 per-activity distributions all inherit the classifier's errors, and no validation on the study's own data is reported. This matches the reader's weakest assumption. A CONDITIONAL verdict remains appropriate: the study would become ACCEPT if the authors provide a validated confusion matrix or corrected estimates, and would weaken if the corrected feature share is no longer dominant. I would keep the reader's CONDITIONAL verdict, so the recommendation is unchanged. The RQ6 documentation extraction has a similar validation gap, but the feature-development distribution is the anchor of the abstract and thus the primary load-bearing concern.","tokens_in":15402,"tokens_out":5245,"duration_ms":54302,"concrete_test":"Independently label a random sample of 100-150 commits from the 931 prompt-changing commits (stratified by predicted class) using two human annotators and a written rubric for feature/fix/refactor; run the same quantized Llama3 classifier and compute the full confusion matrix. Re-estimate RQ4's percentages by applying the inverse misclassification rates (or report the corrected distribution). If the corrected feature share drops below 50% or the corrected fix/refactor shares change by more than 10 percentage points, the headline claim should be revised or reworded; if the corrected distribution is stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline that prompts evolve 'primarily... during feature development' (Abstract; RQ4, Observation 4) depends on classifying 931 prompt-changing commits into feature/fix/refactor using a 4-bit quantized Llama3 70B (Section IV-E1). The only accuracy evidence cited is prior work's 51% overall, and Section IV-E1 does not report precision, recall, or a confusion matrix for this model on the study's own commit distribution. With three coarse classes, 51% overall leaves substantial room for systematic confusion: if feature commits are preferentially mislabeled as refactors (or vice versa), the reported 59.7% feature share could be a biased estimate, not merely noisy. The paper's Section VI claims a 'thorough manual review' of LLM commit classification, but no procedure, sample size, or agreement statistic is given, so the claim is not independently checkable. Since RQ5's per-activity change-type patterns and the 'feature development emphasis' narrative are downstream of this same labeling, the classifier validation is load-bearing. The manual coding of change types (Jaccard 65%, Section IV-D) and the RQ9 seven-project analysis are also limitations, but they do not by themselves threaten the primary quantitative distribution as directly as the unvalidated commit labels do.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how developer-written prompts evolve in open-source LLM-integrated applications. Using the PromptSet dataset and Git history, the authors collected 1,262 prompt changes across 243 GitHub repositories. They apply qualitative coding to categorize change types and prompt components, use a 4-bit quantized Llama3 70B model to classify prompt-changing commits into maintenance activities, extract and categorize commit messages mentioning prompt changes, and perform manual/LLM-based analyses of logical inconsistencies and of whether prompt changes produce the intended LLM response. The main findings are that prompts evolve primarily through additions and modifications, that most prompt-changing commits occur during feature development (59.7%), that only 21.9% of prompt changes are explicitly documented in commit messages, and that prompt changes can introduce logical inconsistencies and sometimes fail to achieve their intended effect on LLM responses.","tokens_in":15766,"tokens_out":5411,"duration_ms":49007,"significance":"If the findings hold, this is a useful first descriptive baseline for prompt evolution as a software maintenance activity. The paper is transparent about its extraction and coding pipeline, reports inter-rater reliability for the qualitative coding, and makes a concrete contribution by characterizing prompts as a distinct software artifact with measurable change patterns. The use of PromptSet and the manual coding of change types are strengths. However, the significance is tempered by the fact that several headline quantitative claims rest on LLM-based classifiers whose accuracy on this study's data is not reported, and by the very small sample underlying the RQ9 conclusion.","major_comments":[{"comment":"The headline distribution of prompt-changing commits across maintenance activities (feature 59.7%, fix 18.9%, refactor 14.5%) is produced by a 4-bit quantized Llama3 70B classifier whose only reported accuracy is the 51% figure from prior work (Section IV-E1). The paper does not report precision, recall, or a confusion matrix for this model on the 931 commits in this study, and the \"thorough manual review\" mentioned in Section VI is not described with any procedure, sample size, or agreement statistic. Because RQ5 and Observation 4 are downstream of this same labeling, the missing validation is load-bearing; systematic confusion between feature and refactor commits, for example, could change the 59.7% estimate substantially. The authors should provide a validation set with per-class F1 or a confusion matrix on a random sample of this study's commits and discuss the sensitivity of the RQ4/RQ5 conclusions to classification error.","section":"IV-E1, V-B (RQ4/Observation 4)"},{"comment":"The 21.9% documentation rate is the basis for the claim that most prompt changes are undocumented. This estimate rests on an LLM-based extraction of commit messages referencing prompt changes (Section IV-F1), but no validation or accuracy measure is reported for this extraction step. False negatives would inflate the undocumented share, while false positives would deflate it. The authors should validate the extraction on a random sample (e.g., manually labeling 100-200 commits) or report precision/recall for the few-shot classifier, and adjust the RQ6 estimate accordingly.","section":"IV-F1, V-C (RQ6)"},{"comment":"The abstract states that \"misalignment often occurs between prompt changes and LLM responses,\" and Observation 9 says prompt modifications \"do not consistently achieve their intended effects.\" This is based on 7 projects (Section V-D), of which 2 showed no response change and 1 showed an unintended change; the other 4 aligned as intended. The paper acknowledges this sample is limited in Section VI, but the abstract presents the finding without the caveat. The authors should either weaken the generalization (e.g., \"in a small sample, we observed...\") or substantially enlarge the set of executable projects before claiming that misalignment is common.","section":"V-D (RQ9), Abstract"},{"comment":"The percentages in RQ4 sum to 93.1% (59.7 + 18.9 + 14.5). The remaining 6.9% of commits is not accounted for. If the LLM classifier produces an \"other\" category or fails to classify some commits, this should be reported explicitly; otherwise the distribution is incomplete and the denominator for the percentages is unclear.","section":"V-B (RQ4)"}],"minor_comments":[{"comment":"The population for the sample size calculation is given as 1,263 while the final dataset is stated as 1,262 (Section IV-C); please confirm which number is correct.","section":"IV-D"},{"comment":"The text says Consideration accounts for 60.4% of \"all prompt changes,\" but the denominator for the component percentages is the sum of component-specific changes (942), not the 1,463 total changes in Table I; rephrase to \"all component-specific changes\" or provide a consistent denominator.","section":"V-A (RQ2, Table II)"},{"comment":"The axes of the pairwise counts heatmap are not defined in the caption; clarify whether rows and columns are change types or component-change pairs and how the proportional frequency is normalized.","section":"Figure 6"},{"comment":"The few-shot prompt used to identify prompt-related commit messages is not shown; including the prompt template would improve reproducibility.","section":"IV-F1"},{"comment":"The claim of a \"thorough manual review\" of the inconsistency-detection LLM results would be easier to assess if the number of cases reviewed and any disagreement counts were reported.","section":"VI (Internal Validity)"},{"comment":"The refined component \"Consideration\" is introduced in Section IV but the reader first encounters it in Section II-B's discussion of the original component set; a brief definition in Section II would improve readability.","section":"II-B, IV-D"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially valuable empirical contribution for an empirical software engineering venue, but the central quantitative claims (RQ4/RQ5 and RQ6) depend on LLM classifiers that are not validated on this study's data. The RQ9 sample is too small for the abstract's claim of frequent misalignment. I would encourage the authors to add classifier validation and to soften the RQ9 generalization; neither issue seems unfixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This is the first study I've seen that traces how developer-written prompts actually change across OSS repositories at scale. The change-type taxonomy (additions/modifications/removals, rephrase/format/correct, etc.) is genuinely useful, and the refinement of the component model—merging Output Formatting and Style into Output Instruction, adding Consideration—looks like a real empirical contribution. The documentation finding (21.9% of prompt changes are explicitly mentioned in commit messages) is clear and reproducible. The inconsistency examples are nice concrete evidence.\n\nThe paper is also transparent about its methods. Manual coding with a codebook and a negotiated agreement, Jaccard 65% given multi-label complexity, is reasonable. And they openly state that RQ9 is based on only 7 projects.\n\nThe soft spot is RQ4/RQ5. The claim that 59.7% of prompt changes occur during feature development is load-bearing, and it rests on a Llama3 70B commit classifier with a reported 51% accuracy from prior work. They don't give precision, recall, or a confusion matrix for this model on their own commit set. With three classes, 51% leaves a lot of room for systematic confusion between feature and refactor labels, and the manual review they mention in Section VI is not described. This doesn't sink the paper—the general direction may well be right—but the specific percentage should be treated as an estimate, not a measurement. The language \"thorough manual review\" needs to be backed with numbers.\n\nTwo smaller quibbles. The 15-word filter is defensible but it will throw away short, legitimate prompts like \"You are a helpful assistant\"—you can't say much about evolution from tiny fragments, but it does limit the scope. And the abstract's \"misalignment often occurs\" is stronger than 3 out of 7 cases supports; that's an exploratory observation, not a prevalence finding.\n\nOverall: this deserves a serious referee. The core descriptive work is solid, the method is mostly transparent, and the topic is important. I'd send it to review with a request that the authors validate the commit classifier on their own data and soften the RQ9 language. It would be a useful addition to the literature after that.","headline":"First large-scale empirical study of prompt evolution in open-source repos, with a genuinely useful change taxonomy, but the headline maintenance-activity split rests on an LLM classifier whose per-class accuracy is never reported.","tokens_in":16187,"tokens_out":2320,"would_cite":true,"duration_ms":22671,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In real-world LLM-integrated applications, prompts behave like a distinct software artifact: they grow mainly by addition and modification during feature work, are rarely documented, and can introduce contradictory instructions.","keywords":["prompt engineering","LLM-integrated applications","prompt evolution","empirical software engineering","software maintenance","commit message documentation","prompt components","logical consistency"],"falsifier":"Take a random sample of the 931 prompt-changing commits, have two independent human experts classify each as feature addition, bug fix, or refactor using the paper's own definitions, and compare against the LLM labels: agreement near the reported 51% overall accuracy would invalidate the maintenance-activity distribution, while agreement above roughly 80% would support it. The same audit applied to the 205 LLM-flagged inconsistency candidates would confirm or refute the 15 manually verified inconsistencies.","tokens_in":15184,"feed_emoji":"🤖","tokens_out":8934,"duration_ms":76118,"temperature":0.7,"pith_summary":"This paper tries to establish that prompts in applications built around large language models (LLMs) are a real, evolving software artifact with measurable maintenance patterns. Analyzing 1,262 prompt changes across 243 open-source repositories, it finds that developers mostly grow prompts by adding new components or modifying existing ones, and that such changes cluster in feature-development work. It also finds that only 21.9% of prompt changes are mentioned in commit messages, that a small but real share of changes introduce logical inconsistencies such as contradictory instructions, and that when a change is executed the LLM sometimes responds exactly as intended, sometimes not at all, and sometimes in an unintended direction. If these results hold, prompt engineering needs its own validation and testing tooling rather than being treated as an ad hoc string-editing task.","feed_headline":"Only 22% of prompt changes get documented in commits","feed_subtitle":"A study of 1,262 prompt changes across 243 repositories shows prompts grow by addition during feature work and can break consistency.","key_machinery":"The load-bearing machinery is a change-extraction and classification pipeline. Starting from an existing dataset of developer-written prompts, the paper cleans it, filters to 243 active repositories, clones each repository, and pairs consecutive versions of prompt-containing files using the ROUGE-L similarity metric to isolate 1,262 distinct prompt changes. It then codes those changes with a refined six-component taxonomy (Directive, Role, Example, Output Instruction, Consideration, Additional Information) and an eight-type change taxonomy (addition, modification, removal, rephrase, formatting, correction, generalization, restructure), reporting a 65% average Jaccard index for inter-rater agreement on multi-label coding. For maintenance-activity questions it classifies commits with a quantized LLM following a prior method whose reported overall accuracy is 51%, and for impact questions it uses chain-of-thought LLM prompting to flag potential inconsistencies, followed by manual review, before attempting to execute changed prompts in the 7 projects with usable tests.","core_discovery":"The central claim is that prompt evolution in real projects follows an expansive, component-focused pattern rather than a corrective or structural one. Developers add new instructions or constraints most often (30.1% of changes) and modify existing ones (25.5%); they remove content only rarely (8.8%). The component most frequently touched, at 60.4% of changes, is the paper's refined 'Consideration' category—instructions and constraints that guide model behavior—while the core 'Directive' that defines the overall task almost never changes. The paper further claims that 59.7% of prompt-changing commits are feature-development commits, that 78.1% of prompt changes go undocumented in commit messages, that prompt changes can produce logical inconsistencies (the paper manually confirms 15 cases from 205 LLM-flagged candidates), and that in the 7 projects that could be executed the LLM response matched the change's intent in 4 cases, stayed unchanged in 2, and deviated in 1. Together these observations define prompts as a new artifact class with its own failure modes.","pith_inferences":["Editorial extension: the 51% accuracy of the prior commit classifier suggests the 59.7% feature-development share should be re-measured on a human-labeled sample before being used as a baseline by later work.","Editorial extension: the paper's prompt-component taxonomy, especially the category it calls 'Consideration,' could be transferred to non-Python projects and to prompts assembled at runtime from many string fragments, where version-to-version pairing is harder.","Editorial extension: the 7-of-101 execution result implies that building prompt regression harnesses with frozen example inputs would let teams detect 'no response change' and 'unintended response change' automatically, turning the paper's observations into testable properties."],"forward_implications":["Because additions and modifications dominate, prompt maintenance tooling should focus on insert-edit operations rather than deletions or structural rewrites.","Since the 'Consideration' component absorbs 60.4% of changes, automated consistency checks could concentrate on newly added instructions and constraints.","With 78.1% of prompt changes undocumented, commit-message generation and prompt-diff documentation tools would address a real traceability gap.","Given confirmed inconsistencies and misaligned LLM responses, prompt changes should be validated by executing representative inputs before merge, not just reviewed as text."],"supporting_citations":[{"why":"Supplies the dataset of developer-written prompts and file paths used to trace prompt versions across commits.","marker":"[41]"},{"why":"Provides the original six-component prompt taxonomy that the study merges and extends into its refined component set.","marker":"[45]"},{"why":"Is the large language model used for commit classification and for extracting prompt-related commit messages.","marker":"[9]"},{"why":"Is one of the prior LLM-based commit-classification approaches whose method and reported accuracy the paper follows.","marker":"[17]"},{"why":"Is the other prior approach used for LLM-based classification of commits into maintenance activities.","marker":"[28]"},{"why":"ROUGE-L is the similarity metric used to detect and measure prompt changes between consecutive versions.","marker":"[33]"},{"why":"Chain-of-thought prompting is the method used to flag potentially inconsistent prompt changes for manual review.","marker":"[52]"},{"why":"Jaccard indexing is the inter-rater reliability measure used to validate the qualitative coding of prompt changes.","marker":"[42]"},{"why":"Establishes the view of prompts as a distinct software artifact, which motivates studying their evolution.","marker":"[31]"}],"fun_headline_variants":["Prompts evolve via addition and tweaks, rarely deletion","Only 22% of prompt edits are documented in commits","Study: prompt changes often break logical consistency","Most prompt edits happen during feature development"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated LLM commit classifier assigns maintenance activities correctly often enough on this dataset; the paper reports only a prior method's overall 51% accuracy and does not give precision and recall for its own 931 commits, so a systematic mislabeling could change the headline feature-development share.","fun_headline_variants_meta":{"raw":{"variants":["Prompts evolve via addition and tweaks, rarely deletion","Only 22% of prompt edits are documented in commits","Study: prompt changes often break logical consistency","Most prompt edits happen during feature development"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3755,"prompt_tokens":951,"completion_tokens":2804,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2744}},"tokens_in":567,"tokens_out":2804,"duration_ms":20188,"temperature":1.0,"reasoning_tokens":2744,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:36:15.003565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 931 prompt-changing commits, have two independent human experts classify each as feature addition, bug fix, or refactor using the paper's own definitions, and compare against the LLM labels: agreement near the reported 51% overall accuracy would invalidate the maintenance-activity distribution, while agreement above roughly 80% would support it. The same audit applied to the 205 LLM-flagged inconsistency candidates would confirm or refute the 15 manually verified inconsistencies.","supporting_citations":[{"cited_title":"Promptset: A programmer’s prompting dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of developer-written prompts and file paths used to trace prompt versions across commits."},{"cited_title":"Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,","cited_arxiv_id":null,"evidence_quote":"Is the other prior approach used for LLM-based classification of commits into maintenance activities."},{"cited_title":"The probabilistic basis of jaccard’s index of similarity,","cited_arxiv_id":null,"evidence_quote":"Jaccard indexing is the inter-rater reliability measure used to validate the qualitative coding of prompt changes."}],"review_version":1}