REVIEW 4 major objections 6 minor 33 references
APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read APIRAT claims that feeding retrieved API sequences and API mappings back into an LLM's translation prompt lifts computational accuracy by 4 to 15.1 percentage points.
desk verdict Useful engineering recipe, but the headline 4-15.1% gains are unproven because the self-test/retry loop is never ablated separately from the API knowledge. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the API knowledge retrieval-and-augmentation loop. Each failed translation triggers retrieval from two vector databases: one holds 200,000 target-language API sequences extracted from popular open-source projects, and the other holds manually curated API mappings (586 Java→Python and 179 Python→Java records) with equivalent target APIs, descriptions, and usage restrictions. Retrieved target sequences are back-translated into source-language API lists so the model can compare structure, and individual source APIs are matched to target mappings to correct single-API usage. These pieces are packed into a knowledge-augmented prompt that asks the model to re-translate; the loop terminates only when the new code passes the generated tests. The design splits API knowledge into sequence-level structure and single-API usage so that the two broad error classes identified in the preliminary study each have a dedicated remedy.
What would settle it
Run the exact APIRAT pipeline but replace the retrieved API sequences and mappings in the second prompt with generic placeholders (or with fixed irrelevant API examples), keeping the same test generation and retry logic. If that control reaches the same computational accuracy as APIRAT, the reported 4–15.1 point gains come from retrying rather than from API knowledge.
Extended reading notes
Core claim
The paper's central claim is that augmenting an LLM with multi-source API knowledge substantially improves cross-language code translation. APIRAT first produces a direct translation and a set of target-language test cases; if execution fails, it retrieves API knowledge and re-prompts the model with that knowledge, repeating until the code passes. The authors report computational accuracy gains over existing LLM-based translation methods of 4 to 15.1 percentage points across Python↔Java translation on CodeNet and AVATAR, with the largest gains on the harder AVATAR Java→Python direction. They also report that the improvement holds across three backbone LLMs and that an ablation attributes positive contributions to each of the three knowledge components, with API sequence retrieval contributing the most.
Load-bearing premise
The load-bearing premise is that the self-test-and-retry loop itself contributes nothing beyond the retrieved API knowledge; the ablation in Table V compares API-knowledge-augmented re-translation against a single direct translation, so there is no control that retries without API knowledge.
Editorial extensions
If this is right
- If the central claim holds, retrieving and injecting API knowledge is a lightweight way to improve LLM code translation without retraining or fine-tuning.
- The reported gains of 4 to 15.1 percentage points over direct and specification-based baselines imply that API mistranslation is a tractable error class, not an inherent LLM limitation.
- Because the method improves all three tested backbone LLMs, including a weaker open model, the approach is likely to transfer to newer LLMs as long as they can follow knowledge-augmented prompts.
- The ablation result that API sequence retrieval contributes the most suggests that preserving call-structure is more valuable than correcting individual API calls, guiding where future knowledge sources should focus.
Reading between the lines
- A natural next experiment the paper does not run is a retry-only control: if retrying without API knowledge already accounts for most of the gain, the method's value would lie in the self-test loop rather than the retrieved content.
- The twelve fault patterns could serve as a reusable diagnostic taxonomy for API-related errors in any LLM code translation system, independent of APIRAT's retrieval design.
- Since the embedding retriever already achieves near-perfect Precision@1 on the constructed benchmark, the remaining headroom is probably in knowledge coverage and prompt integration; testing with oracle API sequences and mappings would reveal how much accuracy is still left on the table.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes APIRAT, an LLM-based code translation method that augments translation prompts with three kinds of API knowledge: retrieved target-language API sequences, back-translated API sequences, and manually curated API mappings. The pipeline first performs direct translation and generates target-language test cases with a rule-based tool; when the translated program fails the generated tests, APIRAT re-translates the program with the retrieved API knowledge added to the prompt. The method is evaluated on CodeNet and AVATAR for Python-to-Java and Java-to-Python translation, reporting computational accuracy improvements of 4-15.1% over existing baselines, generalization across StarCoder, GPT-3.5-Turbo, and GPT-4o-mini, an ablation of the three knowledge components, a retriever comparison on a new APISEQDATA benchmark, and a parameter sensitivity analysis. The main claim is that integrating multi-source API knowledge substantially reduces cross-lingual API mistranslation by LLMs.
Significance. If the attribution to API knowledge is correct, the paper makes a useful contribution: it identifies a concrete failure mode of LLM code translation, designs three relatively inexpensive retrieval-augmented augmentation strategies, manually curates an API mapping pool, releases the tool, and evaluates across multiple models and datasets. The retriever result is striking, with text-embedding-3-large achieving near-perfect Precision@1 on the new benchmark. The ablation also suggests that each proposed component adds value. However, the central comparison conflates API augmentation with the self-test and re-translation loop, so the magnitude and even the direction of the reported gains cannot be attributed to the API knowledge components without an additional control. This is a load-bearing issue for the paper's main claim, though it is addressable with a retry-only baseline.
major comments (4)
- [Section IV-D, Table V] The ablation's 'None' row reports 82.0%/63.5%/49.6%/67.5%, exactly matching the 'Direct translation' row in Table III. This is a single-pass translation without the self-test and re-translation loop that every other row in Table V runs. The comparison therefore conflates two variables: the presence of retrieved API knowledge and the availability of a second test-informed translation attempt. Since self-debugging or re-prompting alone is a known effective repair mechanism (as the paper itself notes via [7]), a retry-only control is essential. The proper baseline is: run the same initial translation, generate test cases, and for failed instances re-translate with the same prompt but without any retrieved API knowledge. Without this control, the 4-15.1% improvement claim and the component contributions in Table V cannot be attributed to API knowledge.
- [Section IV-F, Figure 4] The hyperparameters k (number of API sequences) and n (number of API mappings) are selected via an ablation on CodeNet, and the answer to RQ5 states the optimal values are k=1 and n=5. If these values are then used to produce the CodeNet results in Table V, the reported CodeNet improvements are tuned on the same dataset, and no held-out split or cross-validation is described. If AVATAR uses different values, that is not stated. The paper should describe the tuning protocol explicitly and ideally report results on a separate tuning set or with cross-validation.
- [Section V-B] The Limitations subsection lists model, scope, and metric limitations but does not acknowledge the missing retry-only control. Given that the central claim rests on the contribution of API knowledge, this omitted confound should be stated as a threat to internal validity, and the retry-only baseline should either be added or the claims should be weakened accordingly.
- [Section IV-A, Table VI] The paper does not report statistical significance or confidence intervals for the accuracy differences, and with 200 CodeNet samples and 249-250 AVATAR samples, differences such as 85.5% vs. 86.0% in Figure 4 may be within sampling noise. The terminology 'significant performance improvements' in Section IV-B should be supported by significance testing or at least by explicit variance information.
minor comments (6)
- [Table IV] The parenthetical values such as '+16.5%' and '+54.5%' are relative percentage changes, not absolute percentage-point differences; for example, StarCoder Py-to-Ja goes from 48.5% to 56.5%, which is +8.0 absolute points. Please label the metric in the table caption to avoid confusion.
- [Section III-F, Prompt 3] Prompt 3 refers to 'API Mappings and API Sequence above', but the ablation conditions in Table V do not specify how the prompt is modified when a component is omitted. Providing the exact prompt template for each ablation condition would improve reproducibility.
- [Section IV-A, Table VI] The table is titled 'Accuracy' while Section IV-A defines Precision@1 as the metric for API retrieval; please use consistent terminology throughout.
- [Section IV-A and V-A] The evaluation relies on text-embedding-3-large and GPT-3.5-Turbo with 'default parameter configuration', but no model version or access date is reported. Since these are proprietary, version pinning is important for reproducibility.
- [Figure 4] The figure shows no error bars or repeated-run variation; given the small sample sizes, reporting only point estimates makes it difficult to interpret the small differences (e.g., 85.5% vs. 86.0% in the API-mapping sensitivity plot).
- [Section III-B] The test generation tool is attributed to [17], which is prior work by the same research group. The paper should briefly discuss why this does not introduce systematic bias in the generated test cases, and whether the APISEQDATA benchmark's provenance from PolyHumanEval [17] affects the retriever comparison.
Circularity Check
No circularity in the derivation; the only provenance note is a minor self-citation ([17]) that is not load-bearing, and the retry-only control gap is a validity threat, not a circular reduction.
full rationale
The paper's central claim—that multi-source API knowledge improves LLM code translation—is evaluated on two external benchmarks (CodeNet and AVATAR) with an independent metric (Computational Accuracy). The API sequence databases are built from GitHub repositories and the API mapping pool is manually reviewed against official documentation; neither is defined in terms of the target accuracy. The back-translation and retrieval steps are empirical augmentations, not fitted parameters, and no equation or table in the paper reduces APIRAT's reported accuracy to its own inputs. The only self-citation is [17] (Tao et al., with overlapping authors Gu and Shen), used for the rule-based test-generation tool and as the source of the APISEQDATA benchmark; this is a public tool/benchmark and is not used to compute the main translation results, so it does not make the derivation circular. The missing 'retry-only' control in Table V is a real methodological confound—the 'None' row equals direct single-pass translation rather than the APIRAT pipeline without API knowledge—but a confound is not a circular reduction; it affects causal attribution, not self-definition. The limitations section (V-B) does not flag this control gap, but the absence of a control is a validity concern, not a circularity step.
Assumptions & free parameters
free parameters (2)
- k (number of retrieved API sequences) =
1
- n (number of API mapping pairs) =
5
assumptions (5)
- domain assumption text-embedding-3-large embeddings capture semantic similarity of API sequences and API mapping queries
- domain assumption Tree-sitter AST traversal correctly extracts API call names and parameters
- domain assumption The rule-based test generation tool from Tao et al. [17] produces valid, non-leaking test cases
- domain assumption The manually curated API mapping pool is accurate and covers the test programs
- domain assumption The LLM back-translation step produces correct source equivalents for retrieved target API sequences
Cite this review
Pith. "Pith review of APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs." pith.science (2026). https://pith.science/paper/BZBWQ7RO
@misc{pith2026250414852,
author = {Pith},
title = {Pith review of: APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/BZBWQ7RO}},
note = {Machine review of arXiv:2504.14852}
}
read the original abstract
Code translation is an essential task in software migration, multilingual development, and system refactoring. Recent advancements in large language models (LLMs) have demonstrated significant potential in this task. However, prior studies have highlighted that LLMs often struggle with domain-specific code, particularly in resolving cross-lingual API mappings. To tackle this challenge, we propose APIRAT, a novel code translation method that integrates multi-source API knowledge. APIRAT employs three API knowledge augmentation techniques, including API sequence retrieval, API sequence back-translation, and API mapping, to guide LLMs to translating code, ensuring both the correct structure of API sequences and the accurate usage of individual APIs. Extensive experiments on two public datasets, CodeNet and AVATAR, indicate that APIRAT significantly surpasses existing LLM-based methods, achieving improvements in computational accuracy ranging from 4% to 15.1%. Additionally, our evaluation across different LLMs showcases the generalizability of APIRAT. An ablation study further confirms the individual contributions of each API knowledge component, underscoring the effectiveness of our approach.
Figures
Reference graph
Works this paper leans on
-
[7]
Teaching large language models to self-debug,
X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” in ICLR, 2024
work page 2024
-
[1]
Automated Cobol to Java recycling,
M. Mossienko, “Automated Cobol to Java recycling,” in CSMR, 2003, p. 40
work page 2003
-
[2]
InterTrans: Leveraging transitive intermediate translations to enhance llm-based code translation,
M. Macedo, Y . Tian, P. Nie, F. R. Cogo, and B. Adams, “InterTrans: Leveraging transitive intermediate translations to enhance llm-based code translation,” arXiv preprint arXiv:2411.01063 , 2024
arXiv 2024
-
[3]
Program translation via code distillation,
Y . Huang, M. Qi, Y . Yao, M. Wang, B. Gu, C. Clement, and N. Sun- daresan, “Program translation via code distillation,” arXiv preprint arXiv:2310.11476, 2023
arXiv 2023
-
[4]
CodeGeeX: A pre- trained model for code generation with multilingual benchmarking on HumanEval-X,
Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Li, T. Su, Z. Yang, and J. Tang, “CodeGeeX: A pre- trained model for code generation with multilingual benchmarking on HumanEval-X,” in KDD, 2023, pp. 5673–5684
work page 2023
-
[5]
On the evaluation of neural code translation: Taxonomy and benchmark,
M. Jiao, T. Yu, X. Li, G. Qiu, X. Gu, and B. Shen, “On the evaluation of neural code translation: Taxonomy and benchmark,” in ASE, 2023, pp. 1529–1541
work page 2023
-
[6]
Stelocoder: a decoder-only LLM for multi-language to Python code translation,
J. Pan, A. Sad ´e, J. Kim, E. Soriano, G. Sole, and S. Flamant, “Stelocoder: a decoder-only LLM for multi-language to Python code translation,” CoRR, vol. abs/2310.15539, 2023
arXiv 2023
-
[8]
Z. Tang, M. Agarwal, A. Shypula, B. Wang, D. Wijaya, J. Chen, and Y . Kim, “Explain-then-translate: an analysis on improving program translation with self-generated explanations,” in Findings of EMNLP , 2023, pp. 1741–1788
work page 2023
Show all 33 references
-
[9]
Spectra: Enhancing the code trans- lation ability of language models by generating multi-modal specifica- tions,
V . Nitin, R. Krishna, and B. Ray, “Spectra: Enhancing the code trans- lation ability of language models by generating multi-modal specifica- tions,” arXiv preprint arXiv:2405.18574 , 2024
2024
-
[10]
On the effectiveness of large language models in domain- specific code generation,
X. Gu, M. Chen, Y . Lin, Y . Hu, H. Zhang, C. Wan, Z. Wei, Y . Xu, and J. Wang, “On the effectiveness of large language models in domain- specific code generation,” ACM Transactions on Software Engineering and Methodology, 2024
2024
-
[11]
Lost in translation: A study of bugs introduced by large language models while translating code,
R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in translation: A study of bugs introduced by large language models while translating code,” in ICSE, 2024, pp. 1–13
2024
-
[12]
Starcoder: may the source be with you!
R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023
2023 arXiv
-
[13]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” NeurIPS, vol. 33, pp. 1877–1901, 2020
1901
-
[14]
Unsupervised translation of programming languages,
B. Rozi `ere, M. Lachaux, L. Chanussot, and G. Lample, “Unsupervised translation of programming languages,” in NeurIPS, 2020
2020
-
[15]
Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,
R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al., “Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,” in NeurIPS Datasets and Benchmarks Track , 2021
2021
-
[16]
Avatar: A parallel corpus for java-python program translation,
W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang, “Avatar: A parallel corpus for java-python program translation,” arXiv preprint arXiv:2108.11590, 2021
2021 arXiv
-
[17]
Unraveling the potential of large language models in code translation: How far are we?
Q. Tao, T. Yu, X. Gu, and B. Shen, “Unraveling the potential of large language models in code translation: How far are we?” in APSEC, 2024
2024
-
[18]
Okapi at TREC-3,
S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford, “Okapi at TREC-3,” in TREC, vol. 500-225, 1994, pp. 109– 126
1994
-
[19]
CodeBERT: A pre-trained model for programming and natural languages,
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of EMNLP, 2020, pp. 1536–1547
2020
-
[20]
CodeSearchNet challenge: Evaluating the state of semantic code search,
H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” CoRR, vol. abs/1909.09436, 2019
1909 arXiv
-
[21]
Graphcode {bert}: Pre- training code representations with data flow,
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. LIU, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcode {bert}: Pre- training code representations with data flow,” in ICLR, 2021
2021
-
[22]
UniXcoder: Unified cross-modal pre-training for code representation,
D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “UniXcoder: Unified cross-modal pre-training for code representation,” in ACL, 2022, pp. 7212–7225
2022
-
[23]
Program transformation with Stratego/XT: Rules, strategies, tools, and systems in stratego/xt 0.9,
E. Visser, “Program transformation with Stratego/XT: Rules, strategies, tools, and systems in stratego/xt 0.9,” in Domain-Specific Program Generation, International Seminar , ser. Lecture Notes in Computer Science, vol. 3016, 2003, pp. 216–238
2003
-
[24]
[Online]
c2rust: Migrate C code to Rust. [Online]. Available: https://github.com/ immunant/c2rust
-
[25]
CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understand- ing and generation,
Y . Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understand- ing and generation,” in EMNLP, 2021, pp. 8696–8708
2021
-
[26]
Leveraging automated unit tests for unsupervised code translation,
B. Rozi `ere, J. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample, “Leveraging automated unit tests for unsupervised code translation,” in ICLR, 2022
2022
-
[27]
Code translation with compiler representations,
M. Szafraniec, B. Rozi `ere, H. Leather, P. Labatut, F. Charton, and G. Synnaeve, “Code translation with compiler representations,” in ICLR, 2023
2023
-
[28]
Exploring and unleashing the power of large language models in automated code translation,
Z. Yang, F. Liu, Z. Yu, J. W. Keung, J. Li, S. Liu, Y . Hong, X. Ma, Z. Jin, and G. Li, “Exploring and unleashing the power of large language models in automated code translation,”arXiv preprint arXiv:2404.14646, 2024
2024 arXiv
-
[29]
Enhancing code translation in language models with few-shot learning via retrieval-augmented generation,
M. Bhattarai, J. E. Santos, S. Jones, A. Biswas, B. Alexandrov, and D. O’Malley, “Enhancing code translation in language models with few-shot learning via retrieval-augmented generation,” arXiv preprint arXiv:2407.19619, 2024
2024 arXiv
-
[30]
TRANSAGENT: An LLM-based multi-agent system for code trans- lation,
Z. Yuan, W. Chen, H. Wang, K. Yu, X. Peng, and Y . Lou, “TRANSAGENT: An LLM-based multi-agent system for code trans- lation,” arXiv preprint arXiv:2409.19894 , 2024
2024 arXiv
-
[31]
When language model meets private library,
D. Zan, B. Chen, Z. Lin, B. Guan, Y . Wang, and J.-G. Lou, “When language model meets private library,”arXiv preprint arXiv:2210.17236, 2022
2022 arXiv
-
[32]
Compositional API recommendation for library-oriented code generation,
Z. Ma, S. An, B. Xie, and Z. Lin, “Compositional API recommendation for library-oriented code generation,” in ICPC, 2024, p. 87–98
2024
-
[33]
ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration,
Y . Wang, Y . Zhang, Z. Qin, C. Zhi, B. Li, F. Huang, Y . Li, and S. Deng, “ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration,” arXiv preprint arXiv:2412.05366, 2024
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.