REVIEW 5 major objections 5 minor 19 references
ACT: Bridging the Gap in Code Translation through Synthetic Data Generation & Adaptive Training
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read ACT finetuning lifts open-source code translation accuracy using synthetic data and adaptive training.
desk verdict A coherent synthetic-data finetuning pipeline, but the evaluation shares its data generator with the training set, so the headline 'consistent gains' claim is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the controller-driven ACT loop: synthetic data generation, finetuning, and execution-based evaluation. Data generation expands seed code samples with breadth-based prompts (changing the sample's purpose) and depth-based prompts (adding constraints, complexity demands, or reasoning steps), translates them to the target language, generates unit tests from the source logic plus the target function signature, strengthens those tests with mutation testing (inserting single bugs to see whether the tests catch them), and retains only samples that pass inside Docker. The controller then decides, from evaluation scores and loss trends, whether to continue finetuning, generate more targeted data from failed cases, or stop.
What would settle it
Run the ACT finetuning pipeline, then evaluate the finetuned models on a held-out set of translations whose unit tests were written by humans independently of the ACT pipeline; if the pass@k improvements over the base models disappear or reverse, the reported gains are an artifact of fitting the synthetic test distribution.
Extended reading notes
Core claim
The central claim is that ACT consistently enhances the effectiveness of open-source code translation models. On Java-to-Go, ACT raises DeepSeek-Coder-V2-Instruct's pass@1 from 0.5348 to 0.6248 and Qwen2.5-Coder-7B-Instruct's from 0.4461 to 0.4865, with pass@5 improving in both cases; on C++-to-Rust, DeepSeek's pass@1 rises from 0.4500 to 0.5653 while Qwen's dips slightly to 0.3950 though its pass@5 improves. The paper attributes these gains to the combination of synthetic dataset expansion, mutation-tested unit test generation, Docker-based execution validation, and an LLM-driven controller that iteratively generates targeted data and adjusts finetuning hyperparameters. Human review of a sample of filtered translations confirmed that the pipeline's accept/reject decisions were accurate, and the authors report that applying the data generation pipeline to industry-scale migration projects increased developer acceleration.
Load-bearing premise
The load-bearing premise is that unit tests generated and strengthened by the pipeline are comprehensive and sufficient to judge translation correctness, so both training-data filtering and final pass@k scores rest on that assumption.
Editorial extensions
If this is right
- ACT-finetuned Qwen2.5-Coder-7B-Instruct and DeepSeek-Coder-V2-Instruct beat their base versions on execution-based pass@1 and pass@5 for Java-to-Go, and DeepSeek also improves on C++-to-Rust.
- The controller limits wasted compute by stopping when gains plateau and by generating targeted samples only for failing cases, so the pipeline can be run iteratively without generating a huge dataset up front.
- Because the pipeline needs only a small seed set (150 source-target pairs), it can be directed at project-specific use cases while keeping code inside the organization.
- The authors report that applying the data generation pipeline to industry-scale migration projects increased developer acceleration.
Reading between the lines
- A natural stress test: evaluate ACT-finetuned models on translations with unit tests written by humans outside the ACT pipeline; if the pass@k gains shrink or reverse, the improvements may reflect fitting the synthetic test distribution rather than general translation ability.
- The mutation-testing step could report a concrete adequacy number, such as the fraction of mutants killed after the refinement loop, letting users judge how trustworthy the tests are as an oracle.
- The same controller loop could be extended to fix failing translations instead of only discarding them, turning each failure into targeted training data, which the paper lists as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ACT (Auto-Train for Code Translation), an automated framework for fine-tuning open-source LLMs on code translation tasks. ACT contains a synthetic data generation module that expands seed code samples (breadth/depth), translates them to a target language, generates unit tests with mutation testing, and validates the data via Docker execution. A controller module adaptively decides when to generate more data, continue fine-tuning, or stop, based on evaluation scores and loss trends. The authors report execution-based pass@1 and pass@5 results for fine-tuning Qwen2.5-Coder-7B-Instruct and DeepSeek-Coder-V2-Instruct on Java-to-Go and C++-to-Rust, and claim that ACT 'consistently enhances' the open-source models. They also present a small human evaluation of 30 training samples per task and describe an industry application in the abstract.
Significance. If the central claim were fully supported, ACT would be a useful practical contribution: it shows how to build a synthetic data pipeline with execution-based filtering and adaptive training loops for code translation, which is of interest to practitioners who want to fine-tune open-source models in-house. The paper's strengths include the concrete description of the pipeline, the use of mutation testing to strengthen generated unit tests, and the inclusion of a working appendix example that walks through the data generation process. However, the empirical evidence in the paper is not sufficient to establish the central claim. The evaluation set is generated by the same LLM-based pipeline used to produce training data, no independent benchmark is used, and one of the four reported pass@1 comparisons shows a regression. These issues weaken the significance of the reported results and the strength of the conclusions that can be drawn.
major comments (5)
- [Section 2.1.4, Section 3, Table 2] The evaluation test set is produced by the same synthetic data generation pipeline that creates the training data. Section 2.1.4 describes a single pipeline that expands seed code from HumanEval, translates it, validates it against generated unit tests, and splits the result into training and test samples. Section 3 states that GPT-4o and Claude-3.5-Sonnet were used for data generation. Consequently, Table 2 measures performance on the same distribution on which the model was fine-tuned, with a test oracle also generated by the same LLM family. This creates a circularity risk: reported gains may reflect fitting to the generator's test expectations rather than improved translation ability on genuinely unseen problems. The paper provides no independent held-out benchmark (e.g., HumanEval-X or MBXP) to break this circularity. The only human evaluation (Section 3, Table 1) covers 30 training samples per task, not the test set, so it does not validate the evaluation data. This is a load-bearing issue for the paper's central claim of 'consistently enhances' and needs to be addressed by re-evaluating on an external benchmark.
- [Table 2, Abstract, Section 4] The claim that ACT 'consistently enhances' open-source models is contradicted by the reported Qwen2.5-Coder-7B-Instruct C++-to-Rust pass@1 result, which drops from 0.4184 (base) to 0.3950 (ACT). The paper acknowledges this as a 'slight decline' and points to a pass@5 improvement, but pass@1 and pass@5 are different metrics, and a regression on pass@1 directly contradicts the 'consistently' wording in the Abstract and the claim in Section 4 that 'ACT... significantly improves code translation accuracy for both models.' The central claim needs to be reformulated or the experiments need to be extended to support a universal claim of consistency.
- [Table 2, Section 4] Table 2 reports point estimates without any confidence intervals, bootstrap estimates, or number of test problems. With only two translation tasks and two models, the differences—e.g., Qwen2.5-Coder-7B-Instruct Java-to-Go pass@1 0.4461 to 0.4865—may be within sampling noise. Without a measure of statistical uncertainty, the reader cannot assess whether the observed improvements (and the one regression) are meaningful. The paper should report standard errors or confidence intervals for pass@k, as well as the number of test samples used for each cell.
- [Section 2.3, Figure 8] Section 2.3 states: 'The unit test cases are assumed to be comprehensive and sufficient due to mutation testing that tests every aspect of the translated code.' This assumption is load-bearing for both data filtering and the final pass@k evaluation, yet the paper provides no quantitative evidence for it. No mutation score, test coverage percentage, or number of mutants created per sample is reported. The appendix example (Figure 8) shows only a single mutant being used to add a few threshold-related test cases. Furthermore, the unit tests are generated by LLMs from the source code and the function declaration of the translated code, so they could be biased by the generator's interpretation of the source logic. The paper should report actual mutation testing statistics or otherwise qualify its claims of test comprehensiveness.
- [Section 3, Section 2.1.1] There is an inconsistency in the reported dataset sizes. Section 3 states that the experiment began with 150 seed source-target pairs and that for each seed sample one breadth-expanded and four depth-expanded variations were generated, which would yield 750 samples. Later in the same section it says '450 samples on average for initial training.' If the number is 450 because of filtering or because only a subset of seeds was expanded, this should be stated explicitly. The discrepancy affects the reproducibility of the experiments and the interpretation of the controller's adaptive data generation decisions.
minor comments (5)
- [Section 2.1.4] Typo: 'Te translated code is executed' should be 'The translated code is executed'.
- [Section 2.1.1, Figure 4] The diverse factor is described as ranging from 1 to 10 in Section 2.1.1, but Figure 4 shows 'Diverse Data Factor [0-10]'. The inconsistency should be resolved.
- [Figure 3] Figure 3 has no confidence intervals or error bars, and the y-axis begins at 0.55, which visually exaggerates the differences between stages. The figure should include error bars if available, and the y-axis should be chosen to avoid misleading scaling.
- [Section 2.4] The Controller is described as using an 'LLM-based decision mechanism,' but no details are given about the prompts, thresholds, or decision logic used to select between continuing training, generating more data, or stopping. This makes the controller's behavior non-reproducible. Please provide a concrete description of the decision procedure.
- [References] The data generation uses GPT-4o and Claude-3.5-Sonnet, but no citations are provided for these closed models. While exact citations may be difficult, generic references to the model cards or technical reports would be useful for completeness.
Circularity Check
Evaluation loop is self-referential: test samples and the unit-test oracle are produced by the same synthetic pipeline, and the controller selects the final model on the same test metric.
-
fitted input called prediction
[Section 2.2 (Finetuning), Section 2.3 (Evaluation), Table 2]
"The finetuning process follows an iterative approach, with evaluations after each training stage to assess model performance. Decisions to continue training, adjust parameters, generate more data, or finalize the experiment with the best-performing model depend on evaluation scores and trends in training and validation loss curves."
The 'evaluation scores' that drive the controller are execution-based pass@k values on the generated test set (Section 2.3). Those same scores decide how many epochs and stages to run, which hyperparameters to try, and which checkpoint is finalized as 'the best-performing model.' The chosen model's scores are then reported in Table 2 as ACT's improvement. The test set is thus used as a model-selection signal during training, so the reported pass@k is a maximum over checkpoints selected on the test criterion rather than an out-of-sample prediction. This is statistical selection on the evaluation metric, making the reported 'prediction' partly forced by fitting to that metric.
-
other
[Section 2.1.4 (Validation of Synthetic Data), Figure 2, Section 3 (Experimental Setup)]
"Only samples that pass all tests are retained for further finetuning and evaluation; those that fail are discarded. ... GPT-4o and Claude-3.5-Sonnet were used for the data generation process. ... The unit test cases are assumed to be comprehensive and sufficient due to mutation testing that tests every aspect of the translated code."
Both the training samples and the test samples come from the same synthetic data-generation loop: seed code is expanded, translated, and validated against unit tests produced by the same LLM family (GPT-4o and Claude-3.5-Sonnet), and Figure 2 explicitly shows 'processing of train and test splits for further model finetuning.' The test labels are those LLM-generated unit tests, so the reported pass@k measures the model's ability to satisfy the generator's test distribution, not an externally fixed ground truth. The only human audit covers 30 generated training samples per task, not the test set, so it does not break the self-referential validation loop.
full rationale
The paper's main derivation chain is: generate synthetic code with GPT-4o/Claude-3.5-Sonnet, filter it by LLM-generated unit tests, finetune an open-source model with the controller choosing stages and hyperparameters from evaluation scores, and then report pass@k on test samples drawn from the same filtered synthetic pipeline. This loop is the load-bearing part of the claimed result. The test set is not an independent external benchmark such as HumanEval-X or MBXP; the authors acknowledge only a 30-sample human check of training data, not of test data. The reported pass@k values are real execution scores and the C++-to-Rust Qwen pass@1 regression is disclosed, so the circularity is not a formula identity or a fully forced result. It is a partial circularity: the evaluation metric and the training signal share the same synthetic generator and the same test-based selection process, which makes the 'consistently enhances' conclusion less independent than presented. Score 6 reflects one or more reported predictions that reduce by construction to fitting the synthetic test distribution, without making the entire derivation vacuous.
Assumptions & free parameters
free parameters (1)
- Diverse factor =
1
assumptions (3)
- domain assumption LLM-generated unit tests, after mutation testing, are comprehensive and sufficient to validate translation correctness.
- domain assumption Synthetic source-target code pairs produced by GPT-4o and Claude-3.5-Sonnet and filtered by execution are high-quality training data.
- domain assumption The automatically selected training hyperparameters are appropriate for the models and datasets.
Cite this review
Pith. "Pith review of ACT: Bridging the Gap in Code Translation through Synthetic Data Generation & Adaptive Training." pith.science (2026). https://pith.science/paper/PVAXRGHW
@misc{pith2026250716478,
author = {Pith},
title = {Pith review of: ACT: Bridging the Gap in Code Translation through Synthetic Data Generation & Adaptive Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/PVAXRGHW}},
note = {Machine review of arXiv:2507.16478}
}
read the original abstract
Code translation is a crucial process in software development and migration projects, enabling interoperability between different programming languages and enhancing software adaptability and thus longevity. Traditional automated translation methods rely heavily on handcrafted transformation rules, which often lack flexibility and scalability. Meanwhile, advanced language models present promising alternatives but are often limited by proprietary, API-based implementations that raise concerns over data security and reliance. In this paper, we present Auto-Train for Code Translation (ACT), an innovative framework that aims to improve code translation capabilities by enabling in-house finetuning of open-source Large Language Models (LLMs). ACT's automated pipeline significantly boosts the performance of these models, narrowing the gap between open-source accessibility and the high performance of closed-source solutions. Central to ACT is its synthetic data generation module, which builds extensive, high-quality datasets from initial code samples, incorporating unit tests to ensure functional accuracy and diversity. ACT's evaluation framework incorporates execution-level checks, offering a comprehensive assessment of translation quality. A key feature in ACT is its controller module, which manages the entire pipeline by dynamically adjusting hyperparameters, orchestrating iterative data generation, and finetuning based on real-time evaluations. This enables ACT to intelligently optimize when to continue training, generate additional targeted training data, or stop the process. Our results demonstrate that ACT consistently enhances the effectiveness of open-source models, offering businesses and developers a secure and reliable alternative. Additionally, applying our data generation pipeline to industry-scale migration projects has led to a notable increase in developer acceleration.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Evaluating large language models trained on code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021
arXiv 2021
-
[2]
Effective test generation using pre-trained large language models and mutation testing
Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Desmarais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171: 0 107468, 2024
work page 2024
-
[3]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
arXiv 2021
-
[4]
Qwen2.5-coder technical report
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186, 2024
arXiv 2024
-
[5]
Phrase-based statistical translation of programming languages
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM international symposium on new ideas, new paradigms, and reflections on programming & software, pp.\ 173--184, 2014
work page 2014
-
[6]
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352, 2024
arXiv 2024
-
[7]
Automatic algorithm specification to source code translation
Suvam Mukherjee and Tamal Chakrabarti. Automatic algorithm specification to source code translation. Indian Journal of Computer Science and Engineering (IJCSE), 2 0 (2): 0 146--159, 2011
work page 2011
-
[8]
Migrating code with statistical machine translation
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Migrating code with statistical machine translation. In Companion Proceedings of the 36th International Conference on Software Engineering, pp.\ 544--547, 2014
work page 2014
Show all 19 references
-
[9]
Statistical machine translation outperforms neural machine translation in software engineering: Why and how
Hung Phan and Ali Jannesari. Statistical machine translation outperforms neural machine translation in software engineering: Why and how. In Proceedings of the 1st ACM SIGSOFT International Workshop on Representation Learning for Software Engineering and Program Languages, pp....
2020
-
[10]
Zero: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020
2020
-
[11]
Unsupervised translation of programming languages
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in neural information processing systems, 33: 0 20601--20611, 2020
2020
-
[12]
Code llama: Open foundation models for code
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023
2023 arXiv
-
[13]
Better together? an evaluation of ai-supported code translation
Justin D Weisz, Michael Muller, Steven I Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. Better together? an evaluation of ai-supported code translation. In Proceedings of the 27th International Conference on Intelligent User...
2022
-
[14]
Wizardlm: Empowering large language models to follow complex instructions
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023
2023 arXiv
-
[15]
Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024
2024 arXiv
-
[16]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[17]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[18]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[19]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.