Pith. sign in

REVIEW 1 major objections 2 minor 1 references

From Custom Logic to APIs: Understanding and Recommending API Replacement Refactorings

T0 review · 1 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hybrid tool called AKIRA recommends replacing custom code with API calls at 90 percent recall by combining pattern rules and a knowledge base.

desk verdict The paper gives a solid first empirical look at API replacement refactorings and shows AKIRA beating prior tools on RETIWA, but the internal 90/88 numbers rest on the same 366 cases used to build the patterns. read the letter →

arxiv 2606.06912 v1 pith:NUAUDMIM submitted 2026-06-05 cs.SE

Bridget Nyirongo , Yanjie Jiang , Yuxia Zhang , Hui Liu This is my paper
classification cs.SE
keywords APIreplacementrefactoringsoftwareempiricalstudyrecommendationsystemJavapatternmatchingknowledgebasecodemaintenance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper mines 166299 commits from six Java projects and validates 366 real instances of developers swapping custom logic for API equivalents. From these it extracts recurring patterns and builds AKIRA, a system that first applies deterministic heuristics and then consults a refactoring-aware knowledge base to judge whether a suggestion is practical. The evaluation reports 90 percent recall and 88 percent precision on the authors' own data and raises recall from 21 percent to 81 percent on an external benchmark, showing that static templates alone miss many complex cases. The work therefore claims that mixing rule-based detection with semantic feasibility checking makes automation of these refactorings feasible at scale.

What carries the argument

AKIRA, a hybrid framework that first applies pattern-deterministic heuristics to candidate sites and then consults a refactoring-aware knowledge base to assess practical feasibility of recommending an API replacement.

What would settle it

Applying AKIRA unchanged to a fresh collection of at least three additional Java projects and finding that recall falls below 70 percent or precision below 70 percent on a new manually validated set would falsify the performance claims.

Watch

Extended reading notes

Core claim

By manually analyzing a curated set of 366 validated API replacement instances drawn from open-source Java commits, the authors characterize the scope and patterns of such refactorings and introduce AKIRA, a hybrid framework that integrates pattern-deterministic heuristics with a refactoring-aware knowledge base to recommend replacements of multi-statement custom logic by equivalent API calls, achieving 90 percent recall and 88 percent precision on the curated dataset and improving recall from 21 percent to 81 percent and precision from 40 percent to 78 percent on the external RETIWA dataset.

Load-bearing premise

The 366 manually validated instances from six open-source Java projects capture the patterns and feasibility conditions that appear in other projects and languages.

Editorial extensions

If this is right

  • Refactoring tools can now handle multi-statement semantic equivalents that simple templates miss.
  • Codebases can be updated to use library APIs more consistently without manual search.
  • Maintenance effort decreases because common replacement opportunities are surfaced automatically during development.
  • The same hybrid pattern-plus-knowledge approach can be reused for other categories of refactoring once additional patterns are collected.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method could be ported to languages other than Java by rebuilding the knowledge base from commits in those languages.
  • Integration into an IDE would let developers accept or reject suggestions in real time rather than after the fact.
  • Expanding the knowledge base with more projects might raise precision further by capturing rarer but still valid patterns.
  • The same feasibility-checking step could be applied to recommend replacements between different versions of the same library.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper reports an empirical study mining 166,299 commits from six Java projects, manually curating 1,800 commits to yield 366 validated API replacement instances, characterizing their patterns, and proposing the AKIRA hybrid framework (pattern-deterministic heuristics plus refactoring-aware knowledge base). It claims 90% recall / 88% precision on the curated dataset and substantial gains over prior work (recall 21%→81%, precision 40%→78%) on the external RETIWA dataset.

Significance. If the internal evaluation demonstrates generalization rather than pattern memorization, the work would be significant: it supplies the first large-scale characterization of API replacement opportunities and a concrete hybrid approach that measurably outperforms existing template-based tools on both internal and external data.

major comments (1)
  1. [Abstract / Evaluation] Abstract (and the evaluation description): the 366 validated instances are explicitly used both to identify recurring patterns and to populate AKIRA’s knowledge base, yet the reported 90%/88% figures are given for performance “on a manually curated dataset” with no mention of a train/test split, cross-validation, or held-out instances. Because the heuristics are derived directly from these instances, the metrics may reflect in-sample matching rather than out-of-sample generalization; this directly affects the strength of the central performance claim.
minor comments (2)
  1. [Empirical study] The sampling procedure that reduced 166,299 commits to the 1,800-commit subset (and then to the 366 labels) is not described; inter-rater agreement statistics for the manual validation are also absent.
  2. [Evaluation] Clarify whether the external RETIWA evaluation used the identical pattern set derived from the internal 366 instances or whether any adaptation occurred.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on the evaluation methodology. We address the major comment below.

read point-by-point responses

  1. Referee: [Abstract / Evaluation] Abstract (and the evaluation description): the 366 validated instances are explicitly used both to identify recurring patterns and to populate AKIRA’s knowledge base, yet the reported 90%/88% figures are given for performance “on a manually curated dataset” with no mention of a train/test split, cross-validation, or held-out instances. Because the heuristics are derived directly from these instances, the metrics may reflect in-sample matching rather than out-of-sample generalization; this directly affects the strength of the central performance claim.

    Authors: We agree that the 366 instances were used both to derive the recurring patterns/heuristics and to populate the knowledge base, and that the reported 90% recall / 88% precision figures are computed on this same curated set without an explicit train/test split or cross-validation. This means the internal metrics are in-sample and do not by themselves demonstrate generalization. The manuscript's primary claim of out-of-sample effectiveness therefore rests on the separate external RETIWA evaluation (recall 21%→81%, precision 40%→78%). We will revise the abstract and evaluation section to explicitly state that the internal figures are in-sample on the full curated dataset, to clarify the dual use of the data, and to position the external results as the main evidence of generalization. We will also investigate whether a k-fold cross-validation experiment on the 366 instances is feasible for the revised manuscript. revision: yes

Circularity Check

1 steps flagged · score 6.0 of 10

Internal 90%/88% metrics reported on the same 366 instances used to extract the pattern-deterministic heuristics

  1. fitted input called prediction [Abstract]
    "we identify 366 validated instances to characterize their scope, categories, and recurring patterns. Based on these insights, we propose AKIRA (Adaptive Knowledge Discovery and Retrieval), a hybrid framework that integrates pattern-deterministic heuristics with a refactoring-aware knowledge base ... Our evaluation shows that AKIRA achieves 90% recall and 88% precision on a manually curated dataset."

    The 366 instances supply both the recurring patterns that define the heuristics and the 'manually curated dataset' on which performance is measured. Because the heuristics are deterministic pattern matchers derived from exactly those instances, the reported recall and precision are forced by construction rather than measuring predictive power on unseen data.

full rationale

The paper extracts recurring patterns from the 366 validated instances, builds AKIRA's pattern-deterministic heuristics and knowledge base directly from those instances, then reports recall/precision on 'a manually curated dataset' with no stated train/test split, cross-validation, or held-out set. This makes the internal performance numbers equivalent to re-detecting the input patterns by construction rather than demonstrating generalization. The external RETIWA results remain independent and are not affected.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claims rest on the representativeness of the six Java projects and the accuracy of the manual labeling step; no free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption Manual inspection of 1,800 commits from the six projects yields an unbiased sample of 366 true API replacement refactorings.
    The 90 % / 88 % figures and the pattern catalog are derived directly from this labeled set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Custom Logic to APIs: Understanding and Recommending API Replacement Refactorings." pith.science (2026). https://pith.science/paper/NUAUDMIM

@misc{pith2026260606912,
  author       = {Pith},
  title        = {Pith review of: From Custom Logic to APIs: Understanding and Recommending API Replacement Refactorings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUAUDMIM}},
  note         = {Machine review of arXiv:2606.06912}
}
read the original abstract

Software refactoring is essential for maintaining code quality. However, API replacement refactoring, which replaces custom logic with API calls, remains underexplored. Existing refactoring tools provide limited support for detecting such opportunities because they rely on predefined templates and have difficulty capturing complex, multi-statement semantic equivalents. To address this limitation, we conduct the first empirical study of API replacement refactorings by mining 166,299 commits across six open-source Java projects and manually analyzing a curated subset of 1,800 commits, from which we identify 366 validated instances to characterize their scope, categories, and recurring patterns. Based on these insights, we propose AKIRA (Adaptive Knowledge Discovery and Retrieval), a hybrid framework that integrates pattern-deterministic heuristics with a refactoring-aware knowledge base to assess the practical feasibility of recommending API replacement refactorings. Our evaluation shows that AKIRA achieves 90% recall and 88% precision on a manually curated dataset. Furthermore, on the external RETIWA dataset, AKIRA significantly improves the state of the art by increasing recall from 21% to 81% and precision from 40% to 78%. These results demonstrate the effectiveness of combining static pattern matching with semantic reasoning to support the automation of recommending complex API replacement refactorings.

Figures

Figures reproduced from arXiv: 2606.06912 by the authors.

Figure 1
Figure 1. IntelliJ IDEA quick-fix suggestions for Listing [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Procedural Pipeline of the Three-Phase Study [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. API replacements (%). 4.2 Results and Analysis 4.2.1 Frequency and Project Distribution. From 16,677 commits retrieved using refactoring-related keywords, 1,800 were manually inspected, yielding 366 con!rmed API replacement refactorings ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Replacement Scope Distribution [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 5
Figure 5. Figure 5: Motivations for Replacement [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 8
Figure 8. Figure 8: AKIRA: a hybrid framework for API replacement using heuristics and AKDR-L. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 canonical work pages

  1. [1]

    total PZQwhere#:is the total number of methods in library!:Td0.3is the global sampling rateT and

    From Custom Logic to APIs: Understanding andRecommending API Replacement RefactoringsBRIDGET NYIRONGOc–chool of nomputer –cience and —echnologyT meijing tnstitute of —echnologyTnhinaYyNJIE JIyNG∗c–chool of nomputer –oftwareT —ianjin ’niversityT nhinaYUXIy ZHyNGc–chool of nomputer –cience and —echnologyT meijing tnstitute of —echnologyT nhinaHUI LIUc–chool...

    work page arXiv

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.