{"id":"5430d323-7652-45c3-9dee-6401da1ac691","arxiv_id":"2501.15387","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Repaying architectural technical debt in 16 usable Apache codebase items increased average class connectivity, but the effect is negligible and smaller than changes seen in non-ATD files.","lead":"The authors tracked 18 self-admitted architecture technical debt items in Apache projects and measured how class dependencies (FAN-IN and FAN-OUT) changed from when the debt was introduced to when it was repaid. They found that repayment coincided with increased incoming dependencies, but the effect sizes were negligible and non-ATD files showed even larger increases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central FAN-IN increase may be an artifact of comparing unpaired file sets: 44.5% of ATD-affected files were deleted by repayment, and only survivors are measured after, biasing connectivity upward.","rationale":"The reader's weakest assumption focuses on ATD-item validity and lifecycle isolation, and correctly notes that the Non-ATD control shows larger average increases and that the reported effect sizes are negligible. My independent pass found a more direct threat to the central claim: the paper compares FAN-IN and FAN-OUT before and after repayment on different file populations. The paper explicitly says deleted or merged files are excluded from the payment-phase analysis (Section IV), and Table IV reports a 44.5% reduction in ATD-affected file count. A Mann-Whitney U test on unpaired sets cannot separate genuine connectivity growth from selection: if low-FAN-IN files are removed, the survivor-only payment distribution shifts right even when no individual file changes connectivity. This is not speculative; the paper's own worked example shows three files removed, and the discussion of 'file reductions' interprets deletion as refactoring or removal of obsolete code. The Non-ATD control also loses files, so it does not control for this compositional bias; it only shows a similar, albeit smaller, selection effect. Because the central claim is the abstract's causal-sounding 'shift toward centralization,' this methodological gap is the most load-bearing concern. I would keep the reader's CONDITIONAL verdict: the paper has honest reporting and a replication package, but the headline claim requires a paired file-level analysis or a substantial rewrite. If the paired analysis fails to reproduce the increase, the verdict for the central claim should move toward REJECT, while the descriptive and methodological contributions may still stand as a proof of concept.","tokens_in":16198,"tokens_out":4956,"duration_ms":47803,"concrete_test":"Using the replication package (Zenodo 14697268), recompute the RQ1 FAN-IN and FAN-OUT analysis only for files present at both the introduction and payment commits, matched by file path or class identity, using a paired test such as Wilcoxon signed-rank and a paired percent-change summary. If the matched FAN-IN increase is no longer significantly positive or falls below the matched Non-ATD change, the central 'centralization' claim is an artifact of file deletion rather than a genuine effect of ATD repayment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ATD repayment increases FAN-IN by 57.5% rests on an unpaired comparison of non-identical file populations. Section IV and V state that files deleted or merged during repayment have no payment-phase FAN-IN/FAN-OUT values and are excluded, leaving an unpaired dataset analyzed with Mann-Whitney U. Table IV shows the number of ATD-affected files fell from 5,135 at introduction to 3,553 at payment (-44.5%). If deleted files had low connectivity, as is typical of removed or obsolete code, the introduction distribution is inflated with low FAN-IN values, while the payment distribution contains only survivors. This compositional change alone can create a statistically significant shift and a positive average percent change even if no individual surviving file changed connectivity. The paper neither reports a paired analysis on the intersection of files nor accounts for the compositional change. The Non-ATD group also lost 21.26% of files, so the control does not remove this artifact; it merely shows a similar bias. Because the abstract's 'shift toward centralization and increased architectural complexity' is the headline contribution, the unpaired design is load-bearing: if the increase disappears when the same files are compared before and after repayment, the central claim is not supported by the data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the lifecycle of self-admitted architectural technical debt (ATD) by tracing 18 VioMod ATD items from the Li et al. dataset across introduction and repayment, using FAN-IN and FAN-OUT dependency metrics. The authors report that ATD repayment is associated with a 57.5% average FAN-IN increase and a 26.7% FAN-OUT increase, interpreting this as a shift toward dependency centralization and increased architectural complexity, and that ATD files are modified less frequently than Non-ATD files. They also report per-item Mann-Whitney tests, Cliff's delta effect sizes, and partial Spearman correlations with cyclomatic complexity and change counts. The paper provides a replication package and acknowledges in places that the overall effect sizes are negligible.","tokens_in":16490,"tokens_out":7233,"duration_ms":67346,"significance":"If the claims were fully supported, the study would offer a useful empirical account of how ATD repayment alters class-level dependency structure, and its use of a control group and effect sizes would be a methodological strength. The replication package and the worked example are valuable for future studies. However, the headline finding is currently overstated relative to the paper's own numbers: the ATD effect sizes are negligible by the paper's cited thresholds, and the Non-ATD comparison shows larger average increases. The paper's main contribution should be reframed as a small-scale, proof-of-concept demonstration that dependency metrics can trace ATD lifecycles, with a cautionary null result about the specificity and magnitude of connectivity changes.","major_comments":[{"comment":"The headline claim that 'ATD repayment increased class connectivity, with FAN-IN increasing by 57.5% on average and FAN-OUT by 26.7%, suggesting a shift toward centralization and increased architectural complexity' is not supported by the paper's own statistics. Table IV reports Cliff's delta of 0.1028 for FAN-IN and 0.1213 for FAN-OUT for ATD items, both below the 0.147 threshold that the paper defines as negligible, and the same table shows Non-ATD items with a larger average FAN-IN increase (+89.06%) and a larger effect size (0.2766). The Answer to RQ1 repeats the overstatement. The abstract and RQ1 answer should be revised to report a negligible-to-small, non-ATD-specific shift, unless a direct ATD-versus-Non-ATD comparison of changes is provided that supports a stronger statement.","section":"Abstract and Section V-A (Table IV)"},{"comment":"The central comparison is unpaired: the number of ATD-affected files falls from 5,135 at introduction to 3,553 at payment (-44.5%), and the paper explicitly excludes files deleted or merged during repayment from the payment-phase metrics, then applies Mann-Whitney U to the introduction set versus the survivor set. If deleted files have lower FAN-IN, the aggregate distribution and the +57.5% average can change simply because the composition of the two samples differs, even if no surviving file changes its connectivity. The Non-ATD control does not remove this artifact because it also loses 21.26% of files and is likewise unpaired. The paper should report a paired analysis on the intersection of files, or otherwise model the attrition, before attributing the percentage increase to ATD repayment.","section":"Section IV and footnote 6; Table IV"},{"comment":"The Mann-Whitney U tests pool 5,135 files from only 18 ATD items (and 753 files from 18 Non-ATD items) and treat every file as an independent observation, although files within one ATD item and one project are correlated. This inflates the reported p-values (e.g., FAN-IN p = 1.24e-16, FAN-OUT p = 5.05e-22) and makes 'statistically significant' uninformative; indeed, Table V shows that for most individual ATD items the null hypothesis is not rejected. Significance claims should be based on item-level summaries, mixed-effects models, or cluster-robust tests, and the text should state clearly that the file-level tests are descriptive rather than evidence of a generalizable effect.","section":"Section V-A"},{"comment":"The comparison of modification counts between ATD and Non-ATD files does not control for the length of the observation interval: ATD counts span from introduction to repayment, while Non-ATD counts span from the initial commit to the recorded commit, and these intervals need not be comparable. Without normalizing by interval duration, matching intervals, or modeling change rates, the conclusion that 'ATD-related files experience fewer changes' (Answer to RQ2) may simply reflect shorter observation windows for ATD items. The paper should report the interval lengths or use a rate-based analysis.","section":"Section VI and Table VII"}],"minor_comments":[{"comment":"The text states that the median FAN-IN grew from 0.69 to 2.01, but Figure 1 reports medians of 1.0 and 6.5 for the same worked example; please reconcile these values.","section":"Section IV and Figure 1"},{"comment":"The sentence 'the directional test shows that we cannot reject the H1' should be phrased as rejecting the null hypothesis in favor of the alternative; as written it conflates the two hypotheses.","section":"Section IV"},{"comment":"Cliff's delta does not measure the standardized difference between two means; it measures the probability that a randomly selected observation from one group is larger than one from the other. The thresholds are cited correctly, but the definition should be corrected.","section":"Section IV"},{"comment":"There is a typo: 'ommitting' should be 'omitting'.","section":"Section III, step 6"},{"comment":"The table heading 'IT's' should be 'Its', and the legend 'p-value > 0.05' should use 'p-value >= 0.05' or explicitly state the boundary convention for the 'N' category.","section":"Section V-B and Table VI"},{"comment":"The filtering from 116 to 18 ATD items is substantial, and 10 of 22 issue-tracker VioMod items could not be linked to a commit hash. The paper should briefly discuss whether unlinked items might differ from linked ones, since this attrition affects the representativeness of the final sample.","section":"Section III, 'Selection of Violation of modularity ATD'"}],"recommendation":"major_revision","confidential_remarks":"For the editor: This is a transparently reported empirical study with a replication package, and the authors are appropriately cautious in parts of the discussion. The main problem is that the abstract and RQ1 answer contradict the paper's own effect-size results and control-group comparison, and the unpaired design makes the headline percentage changes uninterpretable as effects of ATD repayment. I believe a major revision can fix these issues by reframing the contribution around the small/negligible effects and the methodological lessons, and by adding paired or otherwise corrected analyses. It may also be worth asking the authors to clarify the provenance of the ATD labels, since the input dataset was produced by a group that includes a co-author of this manuscript; the paper is transparent about its re-labeling, but an independent audit or inter-rater agreement statistic would strengthen the validity discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is applying FAN-IN/FAN-OUT to the lifecycle of self-admitted architectural debt, file by file, from introduction to repayment. They ship a replication package, run per-item analyses, and compare against Non-ATD controls. That is real work and a reasonable framing for a proof of concept.\n\nThe problem is the headline result. The abstract says ATD repayment \"increased class connectivity, with FAN-IN increasing by 57.5%... suggesting a shift toward centralization.\" But the paper's own effect sizes are negligible (Cliff's delta 0.10 for FAN-IN), and the Non-ATD control shows a larger average increase (+89% vs +57.5%). So the specific attribution to ATD repayment is weak.\n\nThere is a deeper issue the stress-test flagged: the comparison is unpaired. Section IV says files deleted or merged during repayment have no payment-phase metrics and are excluded. Table IV shows the ATD sample dropped from 5,135 to 3,553 files (-44.5%). If the removed files were mostly low-connectivity ones — the worked example's deleted files have FAN-IN of 1 or 2 — then the introduction distribution is pulled down by files that never survive to payment, and the measured increase partly reflects composition, not a real change in surviving classes. The Non-ATD control also lost 21% of files, so it does not rescue the design. The authors acknowledge the unpaired data but don't quantify the bias. That is load-bearing.\n\nAlso, the pooled Mann-Whitney treats 5,135 files as independent though they cluster within 16 ATD items; per-item tests mostly show no effect. That is a second statistical issue, related but not identical. The threats-to-validity section honestly mentions some of this, and the paper does call itself a proof of concept. I believe the descriptive measurements are likely accurate, and the replication package will let others re-analyze.\n\nBottom line: this deserves a serious referee, but not acceptance as is. A revision needs a paired analysis on surviving files (or a proper survival-aware model), a tempering of the abstract to match the negligible effect sizes, and clustered tests. For reading group, I'd use it as a case study in how survivor bias can shape empirical SE claims.","headline":"A solid proof-of-concept with a real replication package, but the headland claim is undermined by survivor bias in the unpaired comparison and the abstract overstates the negligible effect sizes.","tokens_in":16963,"tokens_out":2125,"would_cite":false,"duration_ms":18945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repaying architectural debt raises class connectivity, study finds","keywords":["architectural technical debt","self-admitted technical debt","dependency analysis","FAN-IN","FAN-OUT","software maintainability","repository mining","empirical software engineering"],"falsifier":"Run the same FAN-IN and FAN-OUT measurements on a matched set of at least 100 ATD repayment commits and a control set of commits matched on file size and age; if the average FAN-IN increase for ATD files is not larger than the control's, the centralization claim is refuted. Even within the current 18 items, computing the change for files matched by SLOC at introduction would test whether the effect survives size normalization.","tokens_in":15925,"feed_emoji":"🔗","tokens_out":4150,"duration_ms":33028,"temperature":0.7,"pith_summary":"This paper sets out to trace how architectural technical debt (ATD) changes a software system from the moment it is introduced to the moment it is repaid, using class-level dependency counts as the measuring instrument. The authors claim that repaying ATD increases class connectivity: average FAN-IN rises 57.5% and FAN-OUT 26.7% between introduction and payment, which they read as dependency centralization that can add architectural complexity even while improving short-term quality. They also claim ATD-affected files are modified less often than comparable non-ATD files, with an average of 6.98 changes per file versus 16.12. The study is explicitly a proof of concept on 18 carefully validated ATD items, and it reports that the effect sizes are small, with Cliff's delta below 0.15 for both metrics.","feed_headline":"Repaying architecture debt lifts class connectivity, study finds","feed_subtitle":"An 18-item proof of concept ties debt repayment to 57% higher FAN-IN; effect sizes stay small.","key_machinery":"The argument is carried by FAN-IN and FAN-OUT, two dependency metrics that count, respectively, how many classes depend on a given class and how many classes that class depends on. The lifecycle is reconstructed with git blame to find the introduction commit and the payment commit for each of 18 validated \"violation of modularity\" ATD items, then static analysis (Understand) recomputes the metrics at both historical states. Mann-Whitney U tests and Cliff's delta quantify whether the connectivity distributions differ and by how much.","core_discovery":"The central discovery, stated on the paper's own terms, is that the repayment phase of architectural technical debt is visible in dependency structure: files touched by ATD repayment gain incoming dependencies (FAN-IN) by 57.5% on average and outgoing dependencies (FAN-OUT) by 26.7%, with medians shifting upward and Mann-Whitney U tests rejecting identical distributions over the full sample. Because the effect sizes are negligible by Cliff's delta (0.1028 for FAN-IN, 0.1213 for FAN-OUT), the authors describe the structural impact as modest rather than dramatic. They additionally find that ATD files change less often than non-ATD files and that, controlling for lines of code, change frequency correlates positively with FAN-IN and FAN-OUT in ATD files but negatively with cyclomatic complexity.","pith_inferences":["The paper's own Non-ATD control shows a larger average FAN-IN increase (+89.06% vs +57.5%), so the observed centralization may be a general software-evolution effect rather than a consequence of ATD repayment; a matched-pair design on file size and age could separate these explanations.","The filter from 116 candidate items to 57 true positives to 18 locatable VioMod items suggests that textual SATD detection captures only a thin slice of architectural debt, implying that structural analysis must be added to NLP-based methods.","The negative correlation between cyclomatic complexity and change frequency in ATD files could mean that complex debt-ridden code deters developers from modifying it; if replicated, this would give a concrete mechanism for why architectural debt persists."],"forward_implications":["If the claim holds, dependency metrics can expose structural side-effects of ATD repayment that quality-focused metrics miss.","Repaying ATD may centralize dependencies, meaning short-term fixes can create new maintenance hotspots that need continued attention.","ATD files being modified less often suggests debt-affected code is avoided or frozen, potentially hiding structural problems from routine maintenance.","Dependency metrics alone are insufficient for ATD impact assessment and should be complemented with other architectural measures.","The method, if scaled to more projects, offers a way to compare the long-term architectural consequences of different debt-repayment strategies."],"supporting_citations":[{"why":"Supplies the original SATD dataset with 116 ATD items across issue sections, pull requests, and commit messages, which the paper filters and re-validates.","marker":"[10]"},{"why":"Provides the classification framework and ATD indicators (violation of modularity, obsolete technology) used for independent re-labeling and final data selection.","marker":"[23]"},{"why":"Establishes the lifecycle tracking approach of separating introduction and payment moments for technical debt items, which the paper adapts for ATD.","marker":"[6]"},{"why":"Motivates the use of FAN-IN and FAN-OUT as coupling metrics linked to refactoring and maintenance effort.","marker":"[18]"},{"why":"The Understand static-analysis tool is used to compute FAN-IN and FAN-OUT at introduction and payment commits.","marker":"[29]"},{"why":"PyDriller mines git history to trace introduction and payment dates and collect per-file change data.","marker":"[31]"}],"fun_headline_variants":["Debt repayment boosts code dependencies by up to 57%","Repaying technical debt centralizes software architecture","ATD repayment shifts class connectivity: FAN-IN up 57.5%","Debt repayment increases code coupling but with small effect","Self-admitted architectural debt: repayment makes code more central"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 18 filtered items are genuine architectural debt instances and that the git-blame-identified introduction and payment commits actually mark the debt's lifecycle rather than ordinary code evolution; the paper's own Non-ATD control, which shows larger dependency increases, is evidence this premise can fail.","fun_headline_variants_meta":{"raw":{"variants":["Debt repayment boosts code dependencies by up to 57%","Repaying technical debt centralizes software architecture","ATD repayment shifts class connectivity: FAN-IN up 57.5%","Debt repayment increases code coupling but with small effect","Self-admitted architectural debt: repayment makes code more central"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3355,"prompt_tokens":1016,"completion_tokens":2339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2256}},"tokens_in":632,"tokens_out":2339,"duration_ms":15060,"temperature":1.0,"reasoning_tokens":2256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:19:30.673775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same FAN-IN and FAN-OUT measurements on a matched set of at least 100 ATD repayment commits and a control set of commits matched on file size and age; if the average FAN-IN increase for ATD files is not larger than the control's, the centralization claim is refuted. Even within the current 18 items, computing the change for files matched by SLOC at introduction would test whether the effect survives size normalization.","supporting_citations":[{"cited_title":"Automatic identification of self- admitted technical debt from four different sources,","cited_arxiv_id":null,"evidence_quote":"Supplies the original SATD dataset with 116 ATD items across issue sections, pull requests, and commit messages, which the paper filters and re-validates."},{"cited_title":"Identification and remediation of self-admitted technical debt in issue trackers,","cited_arxiv_id":null,"evidence_quote":"Provides the classification framework and ATD indicators (violation of modularity, obsolete technology) used for independent re-labeling and final data selection."},{"cited_title":"The lifecycle of technical debt that manifests in both source code and issue trackers,","cited_arxiv_id":null,"evidence_quote":"Establishes the lifecycle tracking approach of separating introduction and payment moments for technical debt items, which the paper adapts for ATD."},{"cited_title":"Refactoring and its relationship with fan-in and fan-out: An empirical study,","cited_arxiv_id":null,"evidence_quote":"Motivates the use of FAN-IN and FAN-OUT as coupling metrics linked to refactoring and maintenance effort."},{"cited_title":"Understand by scitools,","cited_arxiv_id":null,"evidence_quote":"The Understand static-analysis tool is used to compute FAN-IN and FAN-OUT at introduction and payment commits."},{"cited_title":"Spadini, M","cited_arxiv_id":null,"evidence_quote":"PyDriller mines git history to trace introduction and payment dates and collect per-file change data."}],"review_version":1}