Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs

Balvinder Singh; Dimitris Rentas; Giriprasad Sridhara; Jakub Gajda; Karol Sawicki; Maciej Nalepa; Magnus Sandelin; Sai Krishna; Sourav Mazumdar; Sujoy Roychowdhury

REVIEW 3 major objections 4 minor 12 references

Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that instruction fine-tuning a Mixtral 8x7b model with a prompt containing statically extracted class/method context and two retrieved example code blocks raises method-level F1 for generated Java test code from 0.55 to…

desk verdict A useful industrial recipe for test generation, but the headline F1 gain rests on an unreported train/test split. read the letter →

arxiv 2506.11006 v1 pith:GHRXSEYZ submitted 2025-04-23 cs.SE

Sai Krishna , Balvinder Singh , Sujoy Roychowdhury , Giriprasad Sridhara , Sourav Mazumdar , Magnus Sandelin , Dimitris Rentas , Maciej Nalepa

show 2 more authors

Karol Sawicki Jakub Gajda

This is my paper · ORCID

classification cs.SE

keywords testcodegenerationstaticprogramanalysisinstructionfine-tuningretrieval-augmentedMixtral8x7BF1scorecomponenttestingJava

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a system that turns natural-language test step descriptions into Java test code inside a large industrial codebase. Its central claim is that a Mixtral 8x7b model, instruction fine-tuned on a prompt template that includes statically extracted class/method context and two retrieved example code blocks, reaches a mean method-level F1 of 0.63 on generated test code, up from 0.55 for the base 8x7b and close to the 0.66 of Mixtral 8x22b. The authors argue that simple prompting fails because the model invents methods and signatures absent from the repository, and that the gain comes from grounding the model in the repository's actual API surface plus developer-written exemplars. If the result holds, a much smaller model can approximate the test-generation quality of a larger one when supplied with the right context, with attendant cost and sustainability benefits.

What carries the argument

The carrying mechanism is a custom instruction-fine-tuning prompt template. The template has four parts: a system instruction, a <methods> section built from a static-analysis graph of the repository listing fully qualified class names and public method signatures reachable from the test block's imports and containing class, two exemplar test descriptions paired with their developer-written code blocks retrieved by embedding cosine similarity, and the query test description. During training the ground-truth code block is appended as the target. The model is fine-tuned with parameter-efficient LoRA (rank 256, alpha 512, dropout 0.1) on a context length of 10,000 tokens. The static-analysis context is meant to prevent invented method names, and the two exemplars are meant to convey repository-specific conventions such as logging calls around each test step.

What would settle it

Extract the set of TestBegin descriptions in the evaluation set and check whether any of their ground-truth code blocks, or any of the two RAG-exemplar blocks shown in their prompts, appear in the instruction fine-tuning set; if overlap exists, rerun the comparison on a truly held-out partition and see whether the 0.63 versus 0.55 gap persists.

Watch

Extended reading notes

Core claim

The discovery is that instruction fine-tuning an 8x7b mixture-of-experts model on prompts that combine static program analysis output with two RAG-selected example test blocks improves the conformity of generated test code to developer-written test code, as measured by F1 over method invocations, from a mean of 0.55 to 0.63 while reducing the standard deviation from 0.33 to 0.30. This brings the fine-tuned 8x7b within 0.03 of the much larger 8x22b model's 0.66, which the authors summarize as an average improvement of 8% over the base model. The evaluation compares the set of methods invoked in the generated code block with the set of methods in the ground-truth developer-written test block, treating missing methods as false negatives and extra methods as false positives.

Load-bearing premise

That the test blocks used for F1 evaluation were never seen during instruction fine-tuning, either as ground-truth training targets or as retrieved exemplars in other blocks' prompts; the paper does not describe the train/evaluation split.

Editorial extensions

If this is right

The fine-tuned 8x7b model's mean F1 of 0.63 versus the base model's 0.55 suggests that instruction fine-tuning with program-analysis context can recover most of the gap to a model roughly three times larger.
Simple prompting without context produces unusable test code with nonexistent method names and signatures, so the practical system requires at least static-analysis grounding.
Test blocks containing multiple conditional checks and custom exception calls remain difficult: even when the exception classes are in the prompt context, the model frequently omits those calls.
Because the dependency extraction used for the prompt template can be performed for any programming language, the paper's approach is not limited to Java test code.
Subject to more compute and data, the paper's future-work direction is that larger models or higher PEFT ranks could push the F1 further upward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The F1 metric counts method invocations but not whether the generated control flow or assertions are correct, so the measured closeness to the 8x22b model may overstate how often the generated tests would actually pass; a compile-rate or assertion-level evaluation would test the practical gain.
If the fine-tuned 8x7b closes most of the gap to 8x22b, that suggests the RAG exemplars and static context supply much of what makes the larger model better, so further gains could come from better retrieval or more exemplars rather than from model scale alone.
The reported improvement could be inflated if any evaluation blocks or their RAG exemplars overlap the instruction fine-tuning set; reporting the train/evaluation split and per-block retrieval provenance would settle this concern.
A testable extension is to measure developer edit distance or time-to-accept on generated blocks, which would quantify the productivity benefit hinted at by the paper's initial positive user-group feedback.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a 4-page EASE-style experience report from Ericsson on generating component test code in Java. The pipeline makes sense: static program analysis builds a graph, RAG pulls two similar test blocks as exemplars, and instruction fine-tuning on a Mixtral 8x7b produces method-level F1 of 0.63 vs 0.55 base, close to 8x22b's 0.66. That is a genuinely useful result for people building in-house test generators on a budget.

What's new: most prior work targets unit tests. This applies the same tricks to component test blocks and gives a concrete comparison of a fine-tuned smaller MoE against a larger off-the-shelf one. The honest write-up of failure modes (hallucinated method names, need for RAG because of logging conventions) is valuable. The paper also acknowledges the metric limitation—F1 on method names only—and says future work should consider logic correctness.

Soft spots: the main one is the evaluation split. The paper never reports how the test blocks in Table 2 were chosen, how many there are, or whether they were excluded from IFT training. Showing ground truth only for 'training and validation samples' implies test samples don't get ground truth in the prompt, but it doesn't tell us those test blocks weren't in the training set. If any overlap exists—either as a target or as one of the two RAG exemplars in another block's prompt—the 0.08 gain could be partly memorization. The SDs are 0.30-0.33, far larger than the mean difference, and there are no significance tests or confidence intervals. That's a big caveat for a two-row table. The single proprietary repository also limits external validity, though that's typical for this kind of work.

That said, the central recipe is coherent and the paper is transparent about its limitations. The stress-test concern about split leakage is legitimate and should be raised in review, but it's fixable with a one-paragraph data description. This isn't a case where the whole thesis collapses.

Who it's for: practitioners at the LLM+static-analysis intersection, and researchers collecting industrial evidence. It deserves a serious referee; an editor should send it to review, with a request that the authors report the split, sample size, and ideally a paired test.

Referee Report

3 major / 4 minor

Summary. The paper describes an industrial pipeline for generating Java test code blocks at Ericsson from natural-language test step descriptions. The pipeline combines static program analysis of the repository to expose available classes and methods, retrieval-augmented generation (RAG) to provide two exemplar code blocks, and instruction fine-tuning (IFT) of a Mixtral 8x7b model with LoRA using a custom prompt template. The evaluation compares generated code with developer-written ground truth at the level of method names, reporting mean F1 scores of 0.55 for base Mixtral 8x7b, 0.63 for the fine-tuned 8x7b, and 0.66 for base Mixtral 8x22b. The central claim is that IFT yields an average improvement of 8% over the base 8x7b model and approaches the performance of the much larger 8x22b model.

Significance. If the result holds, the paper provides a useful industrial data point: a moderate-size Mixtral 8x7b model, given static-analysis context and two RAG exemplars, can approach the method-level F1 of Mixtral 8x22b on test code generation, with corresponding cost and sustainability benefits. The paper is honest about the main limitation of the F1 metric, explicitly stating that it does not capture logic correctness, and it identifies future work on continual pre-training and formal user studies. The practical significance is real, but the evaluation as reported is not yet sufficient to support the headline quantitative claim. The evaluation set is not defined, no sample size is given, and the standard deviations are large relative to the mean difference. These gaps are load-bearing because the paper's main contribution is the measured improvement from IFT, not the architecture of the pipeline alone.

major comments (3)

[Section 2.2, Table 2] The paper does not define how the test blocks used in Table 2 were sampled or whether they are disjoint from the instruction fine-tuning set. Section 2.2 states that ground-truth code blocks are provided for 'training and validation samples' but never describes the test set. If any evaluated block also appeared as a training target, the measured 0.08 mean F1 gain could reflect memorization rather than generalization. Please report the total number of test blocks, the train/validation/test split sizes, and the exact procedure used to prevent overlap between training targets and evaluated blocks.
[Section 3.2, Table 2] The reported standard deviations (0.30 to 0.33) are roughly four times the mean difference between Mixtral 8x7b IFT (0.63) and base 8x7b (0.55), and the paper gives no sample size, confidence interval, or significance test. The statement in Section 3.2 that 'IFT leads to a significant improvement' is therefore not supported by the data as reported. A paired comparison on the same test blocks, with the number of blocks and a suitable test (e.g., Wilcoxon signed-rank or paired bootstrap), is needed to substantiate the central claim.
[Section 2.2, Retrieval Augmented Generation] The RAG retrieval step is underspecified with respect to potential leakage. The paper says that similar code blocks are retrieved by identifying TCBDs with high cosine similarity to the query TCBD, but it does not state whether the retrieval pool includes the target block itself or its source file, nor whether retrieved exemplars are filtered to exclude the ground truth. Since two RAG exemplars appear in every prompt, including test prompts, self-retrieval of the target block would directly place the ground-truth code block in the prompt. Please describe the retrieval pool, exclusion criteria, and whether any evaluated prompt contained the target block's own code or a block from the training set.

minor comments (4)

[Abstract and Section 3.2] The phrase 'an average improvement of 8%' is ambiguous: the F1 means differ by 0.08 absolute (0.63 versus 0.55), whereas the relative improvement is about 14.5%. Please state which measure is intended.
[Section 3.2] The notation 'Mixtral 8*7B' differs from the 'Mixtral 8x7B' and 'Mixtral 8x22B' used elsewhere; please standardize the model names.
[Section 5] There are typographical errors such as 'perfomance' in Section 5 and 'superfically' in Section 3.2; please correct them.
[Section 2.2 and 3.1] Please report the inference decoding settings (temperature, top-p, number of samples) and describe how non-compilable or parse-failing outputs are handled in the F1 computation, since these choices can materially affect the reported scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the F1 evaluation is externally defined against developer-written ground truth, and the IFT improvement is not derived from fitted parameters or self-cited premises.

full rationale

The paper's central claim is that instruction fine-tuning of Mixtral 8x7b improves method-level F1 for generated test code from 0.55 to 0.63, close to Mixtral 8x22b at 0.66. The F1 metric is defined in Section 3.1 by comparing method invocations in the generated code against developer-written ground truth obtained via program analysis; it is not defined in terms of the model's own outputs or any fitted parameter. The prompt template (Listing 2) contains context (methods, RAG exemplars) plus the ground truth block only for training/validation samples; the model's output is generated free-form, so no 'prediction' is equal to an input by construction. The only self-citations ([8], [9]) are listed as related work and are not load-bearing premises for the reported IFT result. The absence of a described train/validation/test split could allow data leakage that might inflate the measured gain, and the large SDs make the 0.08 difference statistically uncertain, but these are reproducibility/validity concerns, not definitional circularity. No equation or construction in the paper reduces a claimed result to its own inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-selected training hyperparameters and on unverified domain assumptions about the fidelity of program analysis, retrieval, the F1 proxy, and the evaluation split. These are listed below.

free parameters (5)

LoRA rank (r) = 256
Hand-selected, specified in Table 1 as part of the PEFT configuration.
LoRA alpha = 512
Hand-selected, specified in Table 1.
LoRA dropout = 0.1
Hand-selected, specified in Table 1.
Training context length = 10,000 tokens
Set in Section 2.2 to fit the methods list and examples.
Number of RAG exemplars in prompt = 2
The prompt template in Section 2.2 includes two retrieved code blocks.

assumptions (4)

domain assumption Method-level F1 against developer-written code is a useful proxy for generated test code quality.
The metric ignores logic correctness; the authors acknowledge this in Section 5.
domain assumption The static program analysis graph correctly enumerates all relevant classes, public methods, and signatures.
Prompt content in Section 2.2 depends entirely on this graph, with no reported validation.
domain assumption The custom BERT embedder retrieves truly similar test blocks that help generation.
Retrieval is described in Section 2.2, but no retrieval quality evaluation is provided.
domain assumption Evaluation test blocks are disjoint from fine-tuning training data.
No train/validation/test split is reported; an unstated independence assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs." pith.science (2026). https://pith.science/paper/GHRXSEYZ

@misc{pith2026250611006,
  author       = {Pith},
  title        = {Pith review of: Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHRXSEYZ}},
  note         = {Machine review of arXiv:2506.11006}
}

read the original abstract

We describe test code generation using Large Language Models (LLMs) in Ericsson. Our input is a test step in natural language (English) and our output is code (Java) which accomplishes the test step. We describe how straight forward prompting does not suffice and results in LLM assuming functions and signatures which are not present in the code repository. We then show how we alleviate the problem by a combination of Retrieval Augmented Generation (RAG) along with prompt engineering that expanded the simple prompt with additional contextual information using static program analysis. We then describe further improvements that we obtained by fine-tuning the underlying LLM. The fine tuning is done based on a custom designed prompt template which has pre-dependent classes, their public methods as well two exemplar outputs obtained from RAG. Our results establish that our fine tuned models help improve the correspondence or conformity with the original developer written test code as measured by the traditional metrics of F1-score based on the methods used in the generated code. Fine tuning of a 8x7b Mixture of Experts (MoE) model leads to an average improvement of 8\% over the base model and is comparable to the scores on a much larger 8x22b MoE model.

Figures

Figures reproduced from arXiv: 2506.11006 by the authors.

**Figure 2.** Comparative distribution of F1 scores for different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[1]

Mistral AI. 2023. Mixtral-8x7B-v0.1. https://huggingface.co/mistralai/Mixtral- 8x7B-v0.1. Accessed: 2025-03-11

work page 2023
[2]

Mistral Community. 2024. Mixtral-8x22B-v0.1. https://huggingface.co/mistral- community/Mixtral-8x22B-v0.1. Accessed: 2025-03-11

work page 2024
[3]

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220–235

2023
[4]

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

2024
[5]

Ludvig Lemner, Linnea Wahlgren, Gregory Gay, Nasser Mohammadiha, Jingxiong Liu, and Joakim Wennerberg. 2024. Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes. arXiv preprint arXiv:2409.06416 (2024)

arXiv 2024
[6]

Yihao Li, Pan Liu, Haiyang Wang, Jie Chu, and W Eric Wong. 2025. Evaluating large language models for software testing. Computer Standards & Interfaces 93 (2025), 103942

work page 2025
[7]

Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2024. Multi-language unit test generation using llms. arXiv preprint arXiv:2409.03093 (2024)

arXiv 2024
[8]

Giriprasad Sridhara, Sourav Mazumdar, et al . 2023. Chatgpt: A study on its utility for ubiquitous software engineering tasks. arXiv preprint arXiv:2305.16837 (2023)

arXiv 2023

Show all 12 references

[9]

Sujoy Roychowdhury, Giriprasad Sridhara, A K Raghavan, Joy Bose, Sourav Mazumdar, Hamender Singh, Srinivasan Bajji Sugumaran, and Ricardo Britto
[10]

Prateek Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Xiaofei Ma, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, et al. 2023. Exploring continual learning for code generation models. arXiv preprint arXiv:2307.02435 (2023)

2023 arXiv
[11]

Dovydas Marius Zapkus and Asta Slotkien ˙e. 2024. Unit test generation using large language models: A systematic literature review. Lietuvos magistrant u ˛ informatikos ir IT tyrimai: konferencijos darbai, 2024 m. geguž ˙es 10 d. (2024), 136–144

2024
[2024]

CODS COMADS (2024)

Static Program Analysis Guided LLM Based Unit Test Generation. CODS COMADS (2024)

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Mistral AI. 2023. Mixtral-8x7B-v0.1. https://huggingface.co/mistralai/Mixtral- 8x7B-v0.1. Accessed: 2025-03-11

work page 2023

[2] [2]

Mistral Community. 2024. Mixtral-8x22B-v0.1. https://huggingface.co/mistral- community/Mixtral-8x22B-v0.1. Accessed: 2025-03-11

work page 2024

[3] [3]

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5, 3 (2023), 220–235

2023

[4] [4]

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

2024

[5] [5]

Ludvig Lemner, Linnea Wahlgren, Gregory Gay, Nasser Mohammadiha, Jingxiong Liu, and Joakim Wennerberg. 2024. Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes. arXiv preprint arXiv:2409.06416 (2024)

arXiv 2024

[6] [6]

Yihao Li, Pan Liu, Haiyang Wang, Jie Chu, and W Eric Wong. 2025. Evaluating large language models for software testing. Computer Standards & Interfaces 93 (2025), 103942

work page 2025

[7] [7]

Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2024. Multi-language unit test generation using llms. arXiv preprint arXiv:2409.03093 (2024)

arXiv 2024

[8] [8]

Giriprasad Sridhara, Sourav Mazumdar, et al . 2023. Chatgpt: A study on its utility for ubiquitous software engineering tasks. arXiv preprint arXiv:2305.16837 (2023)

arXiv 2023