{"id":"0771cc64-451d-4c7e-b165-1b6b1725bc31","arxiv_id":"2508.17720","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RepoTransAgent raises average compile and pass rates for Java-C# repository translation to 55.34% and 45.84% on the authors' six-project dataset.","lead":"This paper presents RepoTransAgent, a three-agent system that retrieves similar code, gathers repository context, and iteratively repairs Java or C# translations based on test failures. On six GitHub projects, it reports higher compile and pass rates than three baseline translation methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported translation counts are internally inconsistent: 655 Java→C# translations cannot come from only 627 Java focal methods, undermining the dataset's validity and the headline rates.","rationale":"The reader identified the self-constructed dataset and the validity of the test oracle as the weakest assumption. My stress-test confirms and sharpens this concern: the reported counts in Section V-A and Table III are internally inconsistent with the per-project focal method counts in Table I. Specifically, there are only 627 Java focal methods but 655 reported Java→C# translation tasks, which is arithmetically impossible if each source method is translated once. This inconsistency means the denominators used to compute compile and pass rates are unreliable, directly threatening the central empirical claim. The reader already noted 'internal count inconsistencies' in their rationale and issued a conditional verdict, which is appropriate. My concern does not change the verdict: the paper should not be accepted without releasing the dataset and reconciling these counts. However, it does not warrant outright rejection because the inconsistency could plausibly be a reporting error rather than a fundamental flaw. Thus the verdict remains CONDITIONAL, pending the authors' response to the data-integrity check.","tokens_in":15868,"tokens_out":8737,"duration_ms":83596,"concrete_test":"Write a data-integrity check that parses Table I and the totals in Section V-A/Table III, and for each translation direction asserts that the number of tasks equals the number of aligned source-target pairs. If the Java→C# total of 655 exceeds the 627 available Java focal methods (or the C#→Java total of 627 mismatches the 655 C# focal methods), the dataset construction or the reported arithmetic is incorrect; recompute the compile and pass rates from a corrected, released dataset to see if the headline percentages (55.34% compile, 45.84% pass) still hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RepoTransAgent achieves up to 55.34% compile rate and 45.84% pass rate depends entirely on the correctness of the evaluation dataset and the reported arithmetic. Section V-A states that 627 functions were translated from C# to Java (363 compiled, 299 passed) and 655 functions from Java to C# (306 compiled, 245 passed). These totals exactly match the sums of focal methods in Table I: 627 Java and 655 C# focal methods. This is internally inconsistent: if Java→C# translates Java source to C# targets, the number of tasks cannot exceed 627, the number of available Java focal methods. Yet 655 such tasks are reported. Conversely, C#→Java tasks number 627, fewer than the 655 C# focal methods, implying an unexplained subset. Table III repeats these totals (328+299=627; 410+245=655), so this is not a typo in one sentence. Because compile and pass rates are computed as percentages of these denominators, an inconsistent denominator means the headline percentages may not reflect actual performance. Furthermore, the dataset construction is under-described: the paper claims to follow Methods2Test but does not explain how Java-C# function pairs are aligned, how test cases are matched, or why per-project counts differ between languages. Without a released dataset or a precise pairing algorithm, the empirical comparison to baselines cannot be independently verified. If the dataset alignment or the counting is flawed, the reported improvements over baselines may be artifacts of incorrect denominators.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RepoTransAgent is a multi-agent LLM framework for repository-aware Java-C# code translation. The system decomposes translation into three specialized roles: a RAG agent that retrieves similar source-target pairs and name matches, a context agent that invokes five repository tools to gather class, import, and method information, and a refine agent that generates code, executes tests, reflects on failures, and iteratively corrects errors, all driven by a dynamic prompting scheme. The authors construct a dataset from six dual-language GitHub projects (Table I: 627 Java and 655 C# focal methods), claim to follow Methods2Test, and evaluate against UniTrans, PLTranslation, and a no-agent baseline using compile and pass rates. With DeepSeek V3 on the C#-to-Java direction they report 363 compiled and 299 passed out of 627 tasks (55.34% average compile, 45.84% average pass), and 306 compiled and 245 passed out of 655 tasks for Java-to-C#. Ablations (Table IV) show the context and refine agents matter most, and cross-model experiments (Figs. 6-8) show consistent gains on larger models but smaller gains on 7B/8B models. The paper claims up to 55.34% compile rate and 45.84% pass rate and asserts significant outperformance of the baselines.","tokens_in":16168,"tokens_out":17344,"duration_ms":167835,"significance":"The paper offers a useful engineering-level integration of retrieval-augmented generation, tool-based context gathering, and reflection-based iterative refinement in a multi-agent pipeline, and the design is internally well motivated. The strengths are the clean ablation isolating each agent's contribution (Table IV), the breadth of the cross-model evaluation over six LLMs at two size scales, a concrete case study showing tool-driven repair grounded in repository context (Fig. 4), and an honest threats section that checks for data leakage and reports agent temperatures. If the reported rates survive a corrected and released dataset, the framework would be a practical advance for repository-aware translation. However, the significance is currently conditional: the evaluation rests entirely on the authors' own unreleased six-project dataset, the reported task counts are internally inconsistent with the dataset statistics, there are no statistical tests or repeated runs, and no external repository-level benchmark anchors the claimed state-of-the-art superiority. The claimed improvements are plausible but not yet established at the level the abstract asserts.","major_comments":[{"comment":"The reported translation counts are internally inconsistent with the dataset statistics in Table I. The focal-method column sums are 113+229+134+67+42+42 = 627 Java methods and 114+291+140+35+25+50 = 655 C# methods, yet Section V-A(a) reports 627 functions translated from C# to Java and 655 functions translated from Java to C#. For a paired dataset, a Java-to-C# task requires a Java source, so the direction with 655 tasks cannot be produced from only 627 Java focal methods, and conversely the C#-to-Java direction should not have exactly the Java focal-method count unless sources are defined differently. The per-project percentages in Table II use these same counts as denominators (for example, lucene 49.56% times 113 and 15.79% times 114), and Table III's arithmetic (627-363=264 and 655-306=349) confirms that the 627/655 totals are not isolated typos. Because compile rate and pass rate are defined as proportions of these tasks, and the abstract's headline 55.34% and 45.84% figures are computed from these denominators, the central quantitative claims are not verifiable as reported until the authors specify the exact pairing and alignment algorithm, justify the per-direction counts, and recompute all rates on corrected denominators.","section":"V-A(a), Tables I-III"},{"comment":"The dataset construction and the evaluation oracle are under-specified to the point of irreproducibility. Saying that the dataset follows the established methodology of Methods2Test cannot be literally valid, because Methods2Test aligns Java focal methods with their Java test methods and provides no cross-language pairing procedure; the paper never states how the Java and C# implementations of each project were aligned (by names, signatures, or tests), why the per-project focal counts differ between languages in Table I, or whether each translation pair has both a Java and a C# side. It is also unclear what is executed when computing pass rate: whether the target repository's existing tests, the source-repository tests ported to the target language, or both. In addition, no dataset or code is released, and key hyperparameters are never given: the retrieval top-k for the RAG agent, the maximum tool-invocation iterations for the context agent, and the maximum refinement iterations for the refine agent. These omissions place the entire baseline comparison beyond independent verification.","section":"IV-A, III-B3, III-C1, III-D4"},{"comment":"The claim that RepoTransAgent significantly outperforms the baselines is not supported by the statistical evidence presented. Because the Refine Agent runs at temperature 0.8 to encourage diverse hypotheses (Section VI), outputs are stochastic, yet every rate in Tables II-IV and Figs. 6-8 comes from a single run with no confidence intervals, no repeated-seed experiments, and no significance tests; with six projects the paper does not even perform a paired comparison or report per-project baseline deltas in a way that would support a claim of significance. The Average rows in Table II are also unweighted means of per-project rates, with 55.34% being the mean of the six project-level compile rates rather than the pooled rate of 363/627 = 57.9%, and this convention is never stated. Finally, Section V-C reports that on Llama3.1-8B and Qwen2.5-7B the improvements were considerably smaller and even exhibited decline in some cases (Fig. 7), which materially qualifies the model-agnostic generalizability claim in the abstract; without variance information the robustness conclusions cannot be assessed.","section":"V (RQ-1 to RQ-3), VI"}],"minor_comments":[{"comment":"The first paragraph of Section IV reads 'methodology employ to evaluate' where 'methodology employed' is intended, and similar grammar slips appear elsewhere (for example, Section III-D1 'configure with specialized expertise'), so the manuscript needs a careful copyedit.","section":"IV"},{"comment":"The Average rows should state explicitly that they are unweighted means over the six projects; as computed they differ from the pooled rates (for example, 363/627 = 57.9% versus 55.34% for RepoTransAgent's C#-to-Java compile rate), and the current labeling invites misinterpretation.","section":"Table II"},{"comment":"The abstract's phrasing that the framework 'achieves up to 55.34% compile rate and 45.84% pass rate' presents average rates as if they were upper bounds; the per-project maxima are substantially higher (for example, 73.81% compile rate on quartz), so 'up to' is misleading.","section":"Abstract"},{"comment":"Rectifier [13], the authors' own prior translation system, is cited but never used as a baseline; given the stated goal of outperforming state-of-the-art translation methods, the omission should be justified, for example by noting explicitly that Rectifier does not handle repository context.","section":"VII-A"},{"comment":"Figures 6 and 7 are nearly unreadable because the fonts are small and the value labels are crowded and overlapping (for example, '20.4 20.5'); a cleaner layout with error bars would also mitigate the single-run reporting issue.","section":"Figs. 6, 7"},{"comment":"The data-leakage check based on exact matching after removing whitespace and comments is weak, because a model that memorized a function could still emit it with renamed variables or reformatting; a stricter similarity metric or a held-out verification would be more convincing.","section":"VI"},{"comment":"The set-intersection analyses (90 additional translations, 104 common, 355 union) report absolute counts without stating the underlying task totals or whether these are single-run observations, which limits their interpretability.","section":"Figs. 5, 8"},{"comment":"Table I shows a wide imbalance in per-project focal counts (for example, itext 67/35 and quartz 42/25); since the headline averages are unweighted, the smallest projects have the same influence as poi (229/291), and this should be acknowledged when interpreting the aggregate numbers.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The reported 627/655 counts coincide exactly with the Table I column sums with the direction labels swapped, which suggests either a labeling error in Section V-A(a) or Table II or a genuine mismatch between the constructed pairs and the reported denominators; the editor may want this resolved before further review. The evaluation universe is small and self-constructed (six projects, no external benchmark, no released artifacts), and the novelty relative to existing RAG-plus-tool-use multi-agent systems is incremental; the paper's fit for the journal depends on whether the authors can substantiate the empirical claims after revision. I would also note that the paper cites repository-level translation benchmarks [14], [15] and its own earlier method [13] without comparing against any of them, which weakens the state-of-the-art positioning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RepoTransAgent is a coherent integration of known pieces — RAG, a context-gathering tool set, dynamic prompts, and reflection-based repair — into a three-agent pipeline for repository-level Java↔C# translation. The direction of the result is believable, and the ablation study is a real strength. But there is a counting inconsistency in the reported evaluation that must be resolved before the headline 55.34%/45.84% numbers can be quoted.\n\nWhat the paper does well: the agent design is concrete and well-specified, with five context tools, selective RAG, and structured dynamically updated prompts. The evaluation is broader than most: six real projects, bidirectional translation, four base LLMs, ablations, error-type analysis, and an exact-match check against data leakage. The ablations clearly show that the Context Agent and Refine Agent carry most of the gain, which makes the mechanism story plausible. The RAG-only contribution is modest, and the authors honestly say so.\n\nWhere the soft spots are: the count problem is real. Table I sums to 627 Java and 655 C# focal methods. Section V-A then reports 627 C#→Java translations and 655 Java→C# translations. Since the number of Java→C# tasks cannot exceed the number of Java source methods, the denominators and the direction labels cannot both be right. Table III repeats the same totals, so this is not a one-line typo. The exact percentages may survive a correction, but the paper needs to state clearly which directions used which source sets, and why. Second, no code or data is released, and the dataset construction is underdescribed: Methods2Test is Java-based, yet the paper gives no algorithm for how Java/C# function pairs were aligned or how tests were matched across languages. Third, there are no statistical tests or confidence intervals anywhere; with a stochastic Refine Agent at temperature 0.8, some variance reporting is needed, plus sensitivity analysis for top-k and maximum iterations. Fourth, Rectifier, a closely related method by overlapping authors, is cited but not included as a baseline. That omission should be addressed.\n\nWho this is for: software-engineering researchers working on code translation or LLM agents. It is an empirical systems paper, not a new paradigm, and its practical value depends on the artifacts being released. I would not cite it before the evaluation is cleaned up and the dataset is available.\n\nRecommendation: send to peer review, but with major revisions. The right bar is: fix the denominator inconsistency, release the dataset and code, add Rectifier as a baseline, and add basic significance or variance reporting. With those, the paper would be a solid contribution; without them, the headline numbers are not independently checkable.","headline":"A believable multi-agent code-translation framework with a real counting inconsistency in the reported evaluation; the headline numbers need artifact release and a denominator fix before they can be trusted.","tokens_in":16679,"tokens_out":5056,"would_cite":false,"duration_ms":51612,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-agent LLM framework raises repository-aware Java-C# translation to 55.34% compile and 45.84% pass rates.","keywords":["multi-agent LLM framework","repository-aware code translation","retrieval-augmented generation","iterative refinement","Java-C# translation","reflection-based error correction","code translation agents"],"falsifier":"A concrete check: take a random sample of the dataset's translation pairs, have developers verify each Java-C# pair is semantically equivalent and that the unit tests exercise the translated method, then re-run the framework on the verified subset. If the compile and pass rates fall well below the reported 55.34% and 45.84%, the headline numbers are an artifact of pair alignment or test selection.","tokens_in":15684,"feed_emoji":"🤖","tokens_out":8923,"duration_ms":87913,"temperature":0.7,"pith_summary":"This paper claims that repository-aware code translation, where a function is translated inside a real codebase rather than in isolation, fails with static prompts and plain test feedback, and that decomposing the job into three collaborating LLM agents fixes much of the failure. The agents retrieve similar functions, gather repository context through five selectable tools, and iteratively translate, run tests, reflect on failures, and re-fetch context before fixing. On hundreds of Java-C# function pairs from six open-source projects, the framework reports compile rates up to 55.34% and pass rates up to 45.84%, roughly doubling the baselines' 26-30% compile and 18-28% pass rates. If this holds, it suggests that copying code across ecosystems is mostly a context problem, finding the target repository's substitutes for classes, methods, and imports, and that agent-driven context retrieval is a practical way to solve it.","feed_headline":"Agent trio lifts Java-C# code translation to 55% compile rate","feed_subtitle":"Three specialized agents retrieve context, reflect on failures, and roughly double baseline pass rates.","key_machinery":"The load-bearing object is the three-agent workflow with a dynamic prompt cycle. The RAG Agent decides whether retrieval is needed and, when it is, pulls similar source-target pairs and similarly named functions from vector stores. The Context Agent repeatedly chooses among five tools, get source class info, get target class info, find target imports, find target class info, and find target method body, each returning JSON, until enough context is gathered. The Refine Agent generates the translation, runs the target project's tests, classifies failures into compilation, runtime, functional, or non-terminating errors, reflects on root causes, re-invokes context tools when needed, and produces a corrected version. Static prompt components such as Goals, Guidelines, Example, and Output Format are coupled with dynamic components such as Input, Gathered Context, and Last Command, which is what lets the agent adapt its tool use to the specific function and prior results.","core_discovery":"RepoTransAgent's central claim is that a multi-agent decomposition, with a RAG Agent that retrieves similar functions, a Context Agent that invokes tools to pull class and method information from the target repository, and a Refine Agent that translates, executes tests, reflects on root causes, and re-retrieves context before correcting, materially improves repository-aware code translation over prior LLM translation pipelines. The paper reports that on its Java-C# dataset, the framework reaches an average compile rate of 55.34% for C# to Java and 43.07% for Java to C#, with pass rates of 45.84% and 32.36%, against baseline compile rates of 26.07% to 30.47% and pass rates of 18.59% to 28.16%. Ablation results attribute most of the gain to the Context and Refine agents, with the RAG Agent contributing a smaller but positive effect. The authors also report that the framework improves all tested LLMs, with larger models benefiting more and small models sometimes degrading, and that different models translate complementary sets of functions.","pith_inferences":["A natural extension the paper leaves implicit is applying the same retrieve-context-then-reflect loop to API migration and framework upgrades, where the hard step is also substituting target-ecosystem equivalents rather than rewriting syntax.","The small RAG contribution suggests a testable tweak: loosen the standalone-function gate that skips retrieval and measure whether more similar-function examples help the hard cases.","The dependence on prompt length suggests a testable simplification: compressed, tool-specific prompts might bring the benefit to smaller 7B/8B models, where the paper observes degradation."],"forward_implications":["Because the dominant failure is the 'symbol not found' error, translation success depends on locating the target repository's equivalent classes and methods, so the same agent-tool pattern is a plausible template for other language pairs.","The ablation's largest drop when the Refine Agent is removed implies that iterative test feedback alone is not enough; root-cause reflection plus targeted context re-retrieval is what drives the gain.","The model-size result implies that agentic translation benefits are only accessible above a capability threshold, and smaller models may need simplified prompts to participate.","The intersection analysis implies that different models fail on different functions, so combining model outputs yields more correct translations than any single model."],"supporting_citations":[{"why":"Baseline that iteratively refines translations using test feedback; the paper's pass-rate comparison must beat it.","marker":"[9]"},{"why":"Baseline that uses test execution to locate and fix translation bugs; central comparison for compile and pass rates.","marker":"[8]"},{"why":"Prior LLM translation-with-corrector work whose error-correction idea the paper extends with reflection and targeted re-retrieval.","marker":"[13]"},{"why":"The LLM uniformly used in the main comparison across methods, on top of which the headline rates are measured.","marker":"[20]"},{"why":"Repository-level translation benchmark that defines the setting the paper targets.","marker":"[15]"}],"fun_headline_variants":["Multi-agent framework lifts Java-C# translation to 55% compile","Agent trio for code translation beats solo LLMs, hits 55% compile","Multi-agent LLMs nearly double translation pass rates","Repository-aware agent framework improves code translation up to 55% compile"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the Java-C# function pairs taken from six open-source projects are correctly aligned and that each project's executed unit tests genuinely test the translated function's behavior; if either the pairing or the tests are wrong, the reported compile and pass rates do not measure translation correctness.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent framework lifts Java-C# translation to 55% compile","Agent trio for code translation beats solo LLMs, hits 55% compile","Multi-agent LLMs nearly double translation pass rates","Repository-aware agent framework improves code translation up to 55% compile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2796,"prompt_tokens":1014,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":1709}},"tokens_in":630,"tokens_out":1782,"duration_ms":12740,"temperature":1.0,"reasoning_tokens":1709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:00:43.956204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: take a random sample of the dataset's translation pairs, have developers verify each Java-C# pair is semantically equivalent and that the unit tests exercise the translated method, then re-run the framework on the verified subset. If the compile and pass rates fall well below the reported 55.34% and 45.84%, the headline numbers are an artifact of pair alignment or test selection.","supporting_citations":[{"cited_title":"Lost in translation: A study of bugs introduced by large language models while translating code,","cited_arxiv_id":null,"evidence_quote":"Baseline that uses test execution to locate and fix translation bugs; central comparison for compile and pass rates."}],"review_version":2}