{"id":"4934fe8f-c1ef-45a1-89b2-93e9141fe9c6","arxiv_id":"2504.14852","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"APIRAT improves LLM code translation accuracy by retrieving and injecting API sequence and mapping knowledge, reporting 4-15.1% computational accuracy gains on CodeNet and AVATAR.","lead":"This paper introduces APIRAT, a method that feeds large language models external API knowledge (API sequences, back-translations, and API mappings) when translating code between Java and Python. It reports 4 to 15.1 percentage point accuracy gains over existing LLM translation methods on the CodeNet and AVATAR benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 4-15.1% gains are attributed to API knowledge, but Table V lacks a retry-only control: the 'None' row is direct translation, not the APIRAT self-test/retry pipeline without knowledge, so self-retry alone could explain part or all of the improvement.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: the absence of a retry-only control. The paper's headline number is a comparison of a full pipeline (initial translation + test + failure-triggered retrieval-augmented retranslation) against single-pass baselines. Because the 'None' ablation row is numerically identical to the direct-translation baseline, it cannot separate the effect of API knowledge from the effect of getting a second, test-informed attempt. This is not an accusation of fabrication; it is a standard omitted-control problem. The open-source release and the APISEQDATA retriever comparison are valuable, but they do not resolve the attribution question. The secondary concern about tuning k and n on the CodeNet test set (Section IV-F) is real but less central; a retry-only baseline would directly test the core causal claim. The appropriate verdict remains CONDITIONAL: accept only if the authors add the missing control or explicitly reframe the claim as 'APIRAT with self-test and retry' rather than 'API knowledge is responsible for the gains.'","tokens_in":14038,"tokens_out":4502,"duration_ms":41556,"concrete_test":"Run a retry-only ablation on both CodeNet and AVATAR under the same evaluation setup (GPT-3.5-turbo, same test-generation tool, same pass@1 CA): for every program whose initial translation fails its generated tests, issue the second prompt from Prompt 3 but with the 'Source to Target API Mappings' and API sequence sections removed, keeping only the conversation history and source code. Compare this row with Table III 'Direct translation' and Table V 'All (APIRAT)'. If retry-only CA is close to All (e.g., within 1-2 points), the reported improvements are not evidence for API knowledge; if it remains at or near the 'None' row, the attribution is supported. Report per-direction numbers and, ideally, 95% confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim--that integrating API sequence retrieval, back-translation, and API mapping yields the reported accuracy gains--requires that the self-test and retry loop contributes nothing beyond the retrieved knowledge. This condition is never controlled. In the APIRAT pipeline (Section III-A/B), the initial translation is validated against generated test cases, and 'for instances where the initial translation fails, APIRAT re-translates the program by augmenting it with API knowledge' (Section III-A). The only no-knowledge row in the ablation (Table V, 'None') is 82.0/63.5/49.6/67.5, identical to the 'Direct translation' row in Table III; it is a single-pass translation with no self-test and no second attempt. The comparison therefore conflates two variables: (1) presence of retrieved API knowledge and (2) availability of a second, test-informed translation attempt. Since self-debugging alone is a known effective repair mechanism (the paper itself cites [7]), a retry-only baseline is essential. Without it, the 4-15.1% improvements over baselines and the component contributions in Table V cannot be attributed to API knowledge. The Limitations section (V-B) acknowledges model/scope limitations but does not flag this missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APIRAT, an LLM-based code translation method that augments translation prompts with three kinds of API knowledge: retrieved target-language API sequences, back-translated API sequences, and manually curated API mappings. The pipeline first performs direct translation and generates target-language test cases with a rule-based tool; when the translated program fails the generated tests, APIRAT re-translates the program with the retrieved API knowledge added to the prompt. The method is evaluated on CodeNet and AVATAR for Python-to-Java and Java-to-Python translation, reporting computational accuracy improvements of 4-15.1% over existing baselines, generalization across StarCoder, GPT-3.5-Turbo, and GPT-4o-mini, an ablation of the three knowledge components, a retriever comparison on a new APISEQDATA benchmark, and a parameter sensitivity analysis. The main claim is that integrating multi-source API knowledge substantially reduces cross-lingual API mistranslation by LLMs.","tokens_in":14261,"tokens_out":4676,"duration_ms":44677,"significance":"If the attribution to API knowledge is correct, the paper makes a useful contribution: it identifies a concrete failure mode of LLM code translation, designs three relatively inexpensive retrieval-augmented augmentation strategies, manually curates an API mapping pool, releases the tool, and evaluates across multiple models and datasets. The retriever result is striking, with text-embedding-3-large achieving near-perfect Precision@1 on the new benchmark. The ablation also suggests that each proposed component adds value. However, the central comparison conflates API augmentation with the self-test and re-translation loop, so the magnitude and even the direction of the reported gains cannot be attributed to the API knowledge components without an additional control. This is a load-bearing issue for the paper's main claim, though it is addressable with a retry-only baseline.","major_comments":[{"comment":"The ablation's 'None' row reports 82.0%/63.5%/49.6%/67.5%, exactly matching the 'Direct translation' row in Table III. This is a single-pass translation without the self-test and re-translation loop that every other row in Table V runs. The comparison therefore conflates two variables: the presence of retrieved API knowledge and the availability of a second test-informed translation attempt. Since self-debugging or re-prompting alone is a known effective repair mechanism (as the paper itself notes via [7]), a retry-only control is essential. The proper baseline is: run the same initial translation, generate test cases, and for failed instances re-translate with the same prompt but without any retrieved API knowledge. Without this control, the 4-15.1% improvement claim and the component contributions in Table V cannot be attributed to API knowledge.","section":"Section IV-D, Table V"},{"comment":"The hyperparameters k (number of API sequences) and n (number of API mappings) are selected via an ablation on CodeNet, and the answer to RQ5 states the optimal values are k=1 and n=5. If these values are then used to produce the CodeNet results in Table V, the reported CodeNet improvements are tuned on the same dataset, and no held-out split or cross-validation is described. If AVATAR uses different values, that is not stated. The paper should describe the tuning protocol explicitly and ideally report results on a separate tuning set or with cross-validation.","section":"Section IV-F, Figure 4"},{"comment":"The Limitations subsection lists model, scope, and metric limitations but does not acknowledge the missing retry-only control. Given that the central claim rests on the contribution of API knowledge, this omitted confound should be stated as a threat to internal validity, and the retry-only baseline should either be added or the claims should be weakened accordingly.","section":"Section V-B"},{"comment":"The paper does not report statistical significance or confidence intervals for the accuracy differences, and with 200 CodeNet samples and 249-250 AVATAR samples, differences such as 85.5% vs. 86.0% in Figure 4 may be within sampling noise. The terminology 'significant performance improvements' in Section IV-B should be supported by significance testing or at least by explicit variance information.","section":"Section IV-A, Table VI"}],"minor_comments":[{"comment":"The parenthetical values such as '+16.5%' and '+54.5%' are relative percentage changes, not absolute percentage-point differences; for example, StarCoder Py-to-Ja goes from 48.5% to 56.5%, which is +8.0 absolute points. Please label the metric in the table caption to avoid confusion.","section":"Table IV"},{"comment":"Prompt 3 refers to 'API Mappings and API Sequence above', but the ablation conditions in Table V do not specify how the prompt is modified when a component is omitted. Providing the exact prompt template for each ablation condition would improve reproducibility.","section":"Section III-F, Prompt 3"},{"comment":"The table is titled 'Accuracy' while Section IV-A defines Precision@1 as the metric for API retrieval; please use consistent terminology throughout.","section":"Section IV-A, Table VI"},{"comment":"The evaluation relies on text-embedding-3-large and GPT-3.5-Turbo with 'default parameter configuration', but no model version or access date is reported. Since these are proprietary, version pinning is important for reproducibility.","section":"Section IV-A and V-A"},{"comment":"The figure shows no error bars or repeated-run variation; given the small sample sizes, reporting only point estimates makes it difficult to interpret the small differences (e.g., 85.5% vs. 86.0% in the API-mapping sensitivity plot).","section":"Figure 4"},{"comment":"The test generation tool is attributed to [17], which is prior work by the same research group. The paper should briefly discuss why this does not introduce systematic bias in the generated test cases, and whether the APISEQDATA benchmark's provenance from PolyHumanEval [17] affects the retriever comparison.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The central weakness is the confounding of API augmentation with the retry loop; this is a fixable experimental-design issue rather than a fundamental flaw. I recommend major revision rather than rejection because adding a retry-only baseline, clarifying the tuning protocol, and reporting prompt variants are all within the scope of the manuscript and would directly test the main claim. I would also ask the editor to consider whether the same-group provenance of the test-generation tool and the APISEQDATA benchmark should be treated as a self-evaluation risk."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid engineering effort with a clear recipe for cross-lingual API translation, but the central attribution is not established. The 4-15.1% improvements over baselines are presented as coming from API knowledge, yet every APIRAT condition includes a self-test-and-retry loop while the only no-knowledge control ('None' in Table V) is direct single-pass translation. That comparison conflates two variables: retrieved API knowledge and the availability of a second, test-informed attempt. Since self-debugging alone is known to help (the paper itself cites [7]), a retry-only baseline is essential. Without it, the ablation in Table V and the abstract's headline numbers cannot be attributed to the API knowledge components. The Limitations section acknowledges model/scope issues but does not flag this missing control.\n\nWhat the paper does well: it releases an open-source tool, evaluates on two public benchmarks, tests three backbone LLMs, and includes an ablation. The preliminary fault-pattern study (twelve API mistranslation patterns) is a useful empirical contribution. The retrieval evaluation on APISEQDATA is a reasonable data point, though that benchmark is built from the same group's PolyHumanEval, which creates some circularity but is not fatal. The API mapping pool is manually reviewed, which is real work.\n\nSoft spots beyond the main confound: hyperparameters k=1 and n=5 are selected via ablation on CodeNet and then used for CodeNet headline results, making those numbers optimistic. Sample sizes are 200-250 per cell, so some differences in Tables IV and V may be within noise, and no confidence intervals are reported. The back-translation step uses an LLM to produce source-equivalents, which is fine but means the reported 97-98% retrieval accuracy is for the embedding lookup, not for end-to-end knowledge quality. The 'novel' label is strong: the three techniques resemble Zan et al.'s retrieval-then-coding, TransCoder-style back-translation, and DomCoder-style API recommendation. The integration is new; the pieces are not. Citations to prior work look fair.\n\nRecommendation: send it to peer review. The recipe is plausible and the engineering is real, but the evaluation needs a retry-only baseline, confidence intervals, and a clear statement that the headline gains are an upper bound for the API-knowledge contribution, not an unbiased estimate. For a reader, the paper is worth a skim for the fault-pattern taxonomy and the retrieval comparison, but treat the 4-15.1% numbers with caution.","headline":"Useful engineering recipe, but the headline 4-15.1% gains are unproven because the self-test/retry loop is never ablated separately from the API knowledge.","tokens_in":14854,"tokens_out":2358,"would_cite":false,"duration_ms":22498,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"APIRAT claims that feeding retrieved API sequences and API mappings back into an LLM's translation prompt lifts computational accuracy by 4 to 15.1 percentage points.","keywords":["APIRAT","code translation","API mistranslation","retrieval-augmented generation","API sequence retrieval","API mapping","computational accuracy","large language models"],"falsifier":"Run the exact APIRAT pipeline but replace the retrieved API sequences and mappings in the second prompt with generic placeholders (or with fixed irrelevant API examples), keeping the same test generation and retry logic. If that control reaches the same computational accuracy as APIRAT, the reported 4–15.1 point gains come from retrying rather than from API knowledge.","tokens_in":13795,"feed_emoji":"🔁","tokens_out":7382,"duration_ms":61803,"temperature":0.7,"pith_summary":"APIRAT claims that the dominant failure mode in LLM code translation—cross-lingual API mistranslation—can be sharply reduced by retrieving API knowledge and feeding it back into the translation prompt. The method translates code, runs it against generated test cases, and, when the result fails, retrieves three kinds of API knowledge: target-language API sequences, those sequences back-translated into the source language, and curated single-API mappings. On the CodeNet and AVATAR benchmarks this loop lifts computational accuracy—the fraction of translated programs producing identical outputs on the same inputs—by 4 to 15.1 percentage points over strong LLM baselines. The results matter because the paper first shows that over 60% of LLM translation errors are API-related, so the intervention targets the largest identifiable error source rather than applying generic prompting.","feed_headline":"API retrieval lifts LLM code translation by up to 15.1%","feed_subtitle":"Retrieved API knowledge beats direct and specification-based translation on both benchmarks.","key_machinery":"The load-bearing mechanism is the API knowledge retrieval-and-augmentation loop. Each failed translation triggers retrieval from two vector databases: one holds 200,000 target-language API sequences extracted from popular open-source projects, and the other holds manually curated API mappings (586 Java→Python and 179 Python→Java records) with equivalent target APIs, descriptions, and usage restrictions. Retrieved target sequences are back-translated into source-language API lists so the model can compare structure, and individual source APIs are matched to target mappings to correct single-API usage. These pieces are packed into a knowledge-augmented prompt that asks the model to re-translate; the loop terminates only when the new code passes the generated tests. The design splits API knowledge into sequence-level structure and single-API usage so that the two broad error classes identified in the preliminary study each have a dedicated remedy.","core_discovery":"The paper's central claim is that augmenting an LLM with multi-source API knowledge substantially improves cross-language code translation. APIRAT first produces a direct translation and a set of target-language test cases; if execution fails, it retrieves API knowledge and re-prompts the model with that knowledge, repeating until the code passes. The authors report computational accuracy gains over existing LLM-based translation methods of 4 to 15.1 percentage points across Python↔Java translation on CodeNet and AVATAR, with the largest gains on the harder AVATAR Java→Python direction. They also report that the improvement holds across three backbone LLMs and that an ablation attributes positive contributions to each of the three knowledge components, with API sequence retrieval contributing the most.","pith_inferences":["A natural next experiment the paper does not run is a retry-only control: if retrying without API knowledge already accounts for most of the gain, the method's value would lie in the self-test loop rather than the retrieved content.","The twelve fault patterns could serve as a reusable diagnostic taxonomy for API-related errors in any LLM code translation system, independent of APIRAT's retrieval design.","Since the embedding retriever already achieves near-perfect Precision@1 on the constructed benchmark, the remaining headroom is probably in knowledge coverage and prompt integration; testing with oracle API sequences and mappings would reveal how much accuracy is still left on the table."],"forward_implications":["If the central claim holds, retrieving and injecting API knowledge is a lightweight way to improve LLM code translation without retraining or fine-tuning.","The reported gains of 4 to 15.1 percentage points over direct and specification-based baselines imply that API mistranslation is a tractable error class, not an inherent LLM limitation.","Because the method improves all three tested backbone LLMs, including a weaker open model, the approach is likely to transfer to newer LLMs as long as they can follow knowledge-augmented prompts.","The ablation result that API sequence retrieval contributes the most suggests that preserving call-structure is more valuable than correcting individual API calls, guiding where future knowledge sources should focus."],"supporting_citations":[{"why":"supplies the vanilla-prompting translation template and the bug taxonomy the paper extends into twelve API fault patterns.","marker":"[11]"},{"why":"defines Computational Accuracy, the metric used for all translation comparisons.","marker":"[14]"},{"why":"provides one of the two public evaluation corpora with accompanying test cases.","marker":"[15]"},{"why":"provides the harder parallel evaluation corpus with stricter test cases.","marker":"[16]"},{"why":"provides the rule-based test-generation tool used in self-testing and the source of the retriever benchmark.","marker":"[17]"},{"why":"is the strongest specification-based baseline whose reported accuracy APIRAT exceeds.","marker":"[9]"},{"why":"is the explain-then-translate baseline that APIRAT outperforms by the largest reported margin.","marker":"[8]"},{"why":"supplies an open backbone LLM and an encoder used as a retriever baseline.","marker":"[12]"}],"fun_headline_variants":["API knowledge lifts LLM code translation by up to 15.1%","APIRAT: Adding API knowledge boosts LLM translation accuracy","LLM translation gets 15.1% accuracy boost from API knowledge","API retrieval powers LLM code translation gains up to 15.1%","Multi-source API knowledge elevates LLM code translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the self-test-and-retry loop itself contributes nothing beyond the retrieved API knowledge; the ablation in Table V compares API-knowledge-augmented re-translation against a single direct translation, so there is no control that retries without API knowledge.","fun_headline_variants_meta":{"raw":{"variants":["API knowledge lifts LLM code translation by up to 15.1%","APIRAT: Adding API knowledge boosts LLM translation accuracy","LLM translation gets 15.1% accuracy boost from API knowledge","API retrieval powers LLM code translation gains up to 15.1%","Multi-source API knowledge elevates LLM code translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2321,"prompt_tokens":879,"completion_tokens":1442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1350}},"tokens_in":495,"tokens_out":1442,"duration_ms":10187,"temperature":1.0,"reasoning_tokens":1350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:38:41.907235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact APIRAT pipeline but replace the retrieved API sequences and mappings in the second prompt with generic placeholders (or with fixed irrelevant API examples), keeping the same test generation and retry logic. If that control reaches the same computational accuracy as APIRAT, the reported 4–15.1 point gains come from retrying rather than from API knowledge.","supporting_citations":[{"cited_title":"Lost in translation: A study of bugs introduced by large language models while translating code,","cited_arxiv_id":null,"evidence_quote":"supplies the vanilla-prompting translation template and the bug taxonomy the paper extends into twelve API fault patterns."},{"cited_title":"Unsupervised translation of programming languages,","cited_arxiv_id":null,"evidence_quote":"defines Computational Accuracy, the metric used for all translation comparisons."},{"cited_title":"Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,","cited_arxiv_id":null,"evidence_quote":"provides one of the two public evaluation corpora with accompanying test cases."},{"cited_title":"Unraveling the potential of large language models in code translation: How far are we?","cited_arxiv_id":null,"evidence_quote":"provides the rule-based test-generation tool used in self-testing and the source of the retriever benchmark."},{"cited_title":"Explain-then-translate: an analysis on improving program translation with self-generated explanations,","cited_arxiv_id":null,"evidence_quote":"is the explain-then-translate baseline that APIRAT outperforms by the largest reported margin."}],"review_version":1}