DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production

Bo Jiang; Chao Peng; Jiayi Qi; Jingyi Ren; Xiaoyun Liang

REVIEW 5 major objections 6 minor 28 references

DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DialogAgent claims that replaying real developer behaviors inside an IDE can produce high-fidelity synthetic training data, raising a deployed code assistant's response acceptance rate by 33%.

desk verdict A useful industrial pipeline with a real arithmetic error in its headline efficiency claim and a 33% acceptance-rate number that no controlled experiment supports. read the letter →

arxiv 2412.08069 v1 pith:ZCFUBW4R submitted 2024-12-11 cs.SE cs.AI

Xiaoyun Liang , Jingyi Ren , Jiayi Qi , Chao Peng , Bo Jiang This is my paper · ORCID

classification cs.SEcs.AI

keywords synthetictrainingdatacodequestionansweringsupervisedfine-tuningUIautomationmulti-turndialogueLLM-as-a-judgeIDEassistantproduction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the bottleneck in fine-tuning code assistants is not model size but the scarcity of realistic developer queries, since privacy policies forbid training on real user data. DialogAgent is an automated pipeline that replays representative developer behaviors inside a real IDE plugin, so every synthesized query-response pair carries authentic context: cursor position, selected code, language, locale, and dialog history. The authors report that this data, filtered to top-scored responses by an automated judge, fine-tunes a seed code model to markedly higher usability and perfect-score rates on six code-question-answering tasks, and that deploying the resulting model raised the product's response acceptance rate by 33%. If these numbers hold, the tool offers a way to keep improving code assistants on proprietary code without exposing user data.

What carries the argument

The pipeline is the machinery. A QA Developer Behavior Analyst module examines real Q&A interactions and classifies them along ten behavioral dimensions, including cursor behavior, triggering method, instruction type, programming language, locale, dialog turns, response reference regions, difficulty, and intent, then outputs a data production plan. A chat configuration generator converts that plan into concrete settings such as which file to open, where to place the cursor, and what query to type. A UI automation tool executes those settings inside a real IDE plugin, recording the resulting multi-turn dialogues. A response generator then draws candidate answers from a pool of three code LLMs and selects the best answer per query using an LLM-as-a-judge with rule-based deductions, keeping only 5-point responses for training.

What would settle it

Take a fixed set of queries and split candidate responses into two training sets, one filtered by the automated judge and one filtered by independent human annotators; train the same seed model on each and compare deployment acceptance rates under identical conditions, and if the human-filtered set outperforms the judge-filtered set then the reported gains are partly an artifact of judge bias.

Watch

Extended reading notes

Core claim

The central claim is that synthetic data generation can cross the fidelity gap that separates generic synthetic code datasets from real IDE interactions. DialogAgent builds a ten-dimension behavioral profile of real users, turns that profile into concrete chat configurations, and then physically drives an IDE Q&A plugin through UI automation to produce multi-turn dialogues in a real editor. Candidate answers come from a pool of three code models, and the best answer per query is chosen by a judge that combines LLM scoring, ranking, and rule-based deductions, with only 5-point responses entering the training set. The reported consequence is that a supervised fine-tuned model trained on this corpus beats the seed model on every measured intent, with usability-rate gains of 2 to 55 percentage points depending on intent, while the online acceptance rate of the deployed assistant rose by 33%.

Load-bearing premise

The load-bearing premise is that the automated judge—LLM scoring plus rule-based deductions—reliably identifies the best responses, because the entire training set is filtered to 5-point answers by this judge and the agreement check covers only 600 responses.

Editorial extensions

If this is right

Training data for code Q&A can be produced at 1,440 pairs per instance per day, 4.8 times the throughput of human annotators, without using private user data.
Including multi-turn dialogue data yields absolute gains of roughly 3 points in perfect-score rate and 6 points in usability rate over single-turn-only training.
Randomly generated synthetic data is measurably weaker than behavior-profiled data, confirming that the behavioral analysis contributes to the gains.
The response pool can be improved by fine-tuning its member models on the previous round's selected data, raising the average perfect-score rate by about 8.65 absolute points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 33% acceptance-rate lift comes from an online deployment without a reported controlled A/B isolation, so part of the gain could come from co-launched changes or from the act of refreshing the model.
The same fidelity argument should transfer to other privacy-restricted verticals, such as office-document assistants or data-analysis copilots, wherever an agent can drive the real interface.
The judge's accuracy was measured at 88.47% on a 600-response sample with no confidence interval; a larger evaluation could move that number, which would change the effective quality of the training set.
A direct head-to-head of DialogAgent data versus human-written data at equal volume would separate the contribution of data fidelity from the contribution of data volume.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this for the pipeline, not for the numbers. The core idea—an LLM agent that analyzes real IDE behavior, generates chat configs, drives an actual VS Code plugin via UI automation, and filters responses through an LLM judge plus rule-based deductions—is a sensible, practical engineering contribution. The behavior taxonomy (10 dimensions, 40+ categories) and the design of the judgment system are worth reading. The self-improvement loop on the model pool is also a reasonable approach, and the manual evaluations, while small, are at least described in enough detail to be checked.

That said, the paper has three soft spots, in increasing order. First, the efficiency claim is internally inconsistent: 1,440 divided by 30 is 48, not 4.8. That is not a typo you can ignore, because the abstract and Section IV-B repeat it, and it is one of the two headline results. Second, the 33% acceptance-rate improvement in Section V is an uncontrolled before/after observation: no A/B traffic split, no control model, no measurement window, no baseline rate, and no discussion of concurrent product changes. The sentence reporting it is the entirety of the evidence. The manual evaluations in Section IV are more informative, but they are also based on 300 samples with no significance tests or confidence intervals, and the judge-accuracy check on 600 responses is a useful sanity check, not a causal proof. Third, the paper never compares against existing synthetic-data methods such as OSS-Instruct; the ablation against a random data baseline is a fair start, but the reader is left without a sense of where DialogAgent sits relative to other data-generation approaches.

The citation pattern looks fine, and the paper is honest about some engineering limitations in its final section, though it does not acknowledge the missing causal control on the deployment claim. The work is clearly from a team that has actually built and run this system, and that counts for something. But as written, the central claims are not supported at the level the abstract promises. The 4.8x error alone would make me want a revision, and the 33% claim needs a controlled experiment or a substantial downgrade of the claim.

Who benefits: practitioners working on synthetic SFT data for code assistants, especially industrial teams with IDE-telemetry constraints. It deserves a serious referee because the pipeline is real and the problem is important. My recommendation is to engage with it—send it to review—but with the expectation that the authors must fix the arithmetic, add significance testing or confidence intervals, and either provide a controlled deployment comparison or restate the acceptance-rate claim as anecdotal.

Referee Report

5 major / 6 minor

Summary. The paper presents DialogAgent, an agent-based pipeline for generating synthetic supervised fine-tuning (SFT) data for code question-answering in IDEs. The pipeline uses a developer-behavior analyst to define data production plans, a chat configuration generator to produce repository-level interaction contexts, a UI automation tool to drive a real IDE Q&A plugin, and a multi-model response generator whose outputs are filtered by GPT-4o scoring with rule-based deductions. The authors report that DialogAgent produces 1,440 query-response pairs per day per instance (claimed as 4.8 times human productivity), that SFT on this data improves manually scored Perfect Score Rate and Usability Rate on a 300-sample evaluation, and that deployment of the resulting model improved an online acceptance rate by 33%. The paper also reports ablations for multi-turn dialogue and data increments, plus a consistency check between automated and human scoring.

Significance. If the central claims were fully supported, DialogAgent would be a practically valuable industrial contribution: it offers a privacy-preserving, scalable synthetic data generation loop for code Q&A, with a detailed behavioral taxonomy and a concrete UI automation mechanism. The authors deserve credit for reporting a multi-stage human evaluation, including a judge-accuracy check, and for documenting engineering choices in the prompt designs and scoring rubrics. However, the evidence as presented does not currently establish the headline effectiveness or efficiency claims, and the paper's central deployment result is an uncontrolled observation rather than a controlled comparison.

major comments (5)

[Section V; Abstract; Conclusion] The 33% acceptance-rate improvement is reported as an uncontrolled before/after observation. The text states that 'along with the launch of DialogAgent's SFT model, the online effectiveness indicator—acceptance rate—was improved by 33%,' but it does not report a traffic split, a baseline acceptance rate, a measurement window, a confidence interval, or an accounting of concurrent product changes such as model or UI updates. Since the abstract and conclusion treat this 33% improvement as the central evidence of practical value, the causal attribution to DialogAgent is unsupported. The limitation paragraph at the end of Section V acknowledges only engineering costs and the need for finer classification; it does not acknowledge the missing experimental control.
[Section IV-B; Abstract] The efficiency claim is internally inconsistent. Section IV-B states that human annotators produce 30 pairs per day per annotator and that DialogAgent generates 1,440 pairs per day per instance, then labels the comparison as '4.8 times the productivity of human annotators.' Dividing 1,440 by 30 gives 48, not 4.8. In addition, 'per instance' and 'per annotator' are not commensurable units unless the resource cost of an instance (LLM API calls, compute, maintenance) is specified. The abstract's claim of 'increasing efficiency by 4.8 times' is therefore not supported by the reported numbers.
[Section IV-A; Section IV-D] The central effectiveness comparisons rest on a private evaluation of 300 responses for the main SFT comparison and 600 responses for the judge-accuracy check, with no significance tests, confidence intervals, sample sizes per intent, or description of how the random sample was drawn or blinded. Some per-intent cells in Tables VI, VII, IX, and X appear to be based on very small denominators, such as the seed model's comment-generation PSR of 16.67% and its post-SFT PSR of 78.57%, so the reported gains could be within sampling noise. The Accuracy5 metric in Eq. (1) is a precision-only measure; although recall is reported separately, no overall inter-rater agreement statistic such as Cohen's kappa is given, and the human-scoring protocol itself is not evaluated for consistency.
[Section III-D; Section IV-D] The data-production loop is self-referential in a way that is only partially checked. Candidate responses are selected into the training set by GPT-4o scoring plus rule-based deduction, and the response-generator pool is then fine-tuned on DialogAgent's own selected output via 'self-improvement.' If GPT-4o has systematic preferences for particular response styles, those preferences are amplified in the generator and propagate into the final SFT model. The consistency check in Section IV-D measures only the precision of the judge's 5-point predictions against human scores on 600 responses; it does not establish that the final SFT model's gains survive on a judge-independent benchmark or in a blinded randomized comparison. A concrete test would be to evaluate the final model on a held-out set of human-labeled responses with the judge's scores blinded, and to report agreement beyond precision.
[Section IV-C; Tables IX-X] The ablation results for data increment are not monotonic. For example, DS-33B-Inst-SFT(subset) numerically exceeds DS-33B-Inst-SFT(Ours) on code repair, code generation, and code explanation in Tables IX and X, yet the text claims that the increased data 'can sustain stable effects.' These reversals are not discussed, and they weaken the claim that adding DialogAgent data and multi-turn data consistently improves the model. The paper should either explain the non-monotonic results or soften the corresponding conclusions.

minor comments (6)

[Section IV-C] 'Ww designed' should be 'We designed.'
[Section IV-A] 'DS-33B-Instuct' should be 'DS-33B-Instruct.'
[Section IV-D] 'judement system' should be 'judgment system.'
[Section I] The sentence 'the acceptance rate of our internal is increased by 33%' is grammatically incomplete; 'our internal' should likely be 'our internal model.'
[Section V] The three cursor-behavior percentages (40%, 33%, and 35%) sum to 108%; the paper should clarify whether these are overlapping categories or percentages of different populations.
[Section IV-C] The 'Random' data baseline is described only as 'randomly, incorporating arbitrary programming languages, selected code fragments, and queries'; please specify which components of DialogAgent are removed in this ablation so the comparison is interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed quality gains are evaluated by independent human scoring, and the self-improvement loop is not measured by its own judge.

full rationale

The paper's central chain is an empirical pipeline rather than a formal derivation, so equation-level circularity patterns do not apply. DialogAgent generates synthetic query-response pairs, filters them through a GPT-4o judge plus rule-based deductions, and uses only 5-point responses for supervised fine-tuning. The claimed quality gains in RQ1 and RQ2 are measured through a 21-annotator manual scoring protocol on sampled real queries, independent of the automated judge, so the improvement is not forced by the data-selection criterion. The one place where self-generated data re-enters the system is the response-pool self-improvement step in Section III-D, but the reported PSR improvements in Table XIV are manual-scoring results, not GPT-4o self-scores, so there is no by-construction equivalence between the filter and the evaluation. The 33% acceptance-rate improvement in Section V is an uncontrolled before/after deployment observation lacking an A/B control, confidence intervals, and measurement detail; that is a causal-attribution and reporting weakness, not a circularity. No load-bearing uniqueness theorem, ansatz-by-citation, or renamed known result is invoked. The limitations paragraph acknowledges engineering cost and classification granularity but does not discuss the deployment measurement; that omission is a correctness concern, not circular reasoning.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-tuned thresholds (penalty scores, 5-point cutoff) and domain assumptions about the reliability of an LLM judge and the representativeness of behavioral categories. These are not fitted against an external benchmark and are not independently validated, which limits the strength of the reported results.

free parameters (6)

Penalty deduction scores = 1, 1, 1, 2, 2, 5
Hand-chosen rule-based deductions in Table III that affect which responses receive a 5-point score and thus enter the training set.
Final score threshold for training data = 5
Only responses scoring exactly 5 are used for SFT; this threshold trades dataset size against quality.
SFT epochs = 3
Training hyperparameter following the seed model setup, fixed without sensitivity analysis.
Inference temperature = 0.3
Used during evaluation inference to generate two response variations; affects measured performance.
Inference top-p = 0.95
Sampling parameter used with temperature 0.3 during evaluation.
Number of response generator models = 3
Pool size of DeepSeek models; the paper reports that the pool achieves 79.76% PSR, and adding models is a design choice.

assumptions (4)

domain assumption GPT-4o scoring correlates with human quality
LLM-as-a-judge is used to select final training responses; only a 600-sample Accuracy5 check is reported, with no error bars.
domain assumption Human annotator scores are ground truth
Used for all evaluation metrics; no inter-annotator agreement is reported.
domain assumption Behavioral dimensions derived from online logs are representative
The data production plan is built on 10 dimensions and 40+ categories from Section III-A; there is no demonstration that these cover the full space of real developer interactions.
domain assumption Open-source repositories from BigCode are representative of user code
Queries are asked against randomly selected open-source repos, but proprietary code contexts may differ significantly.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production." pith.science (2026). https://pith.science/paper/ZCFUBW4R

@misc{pith2026241208069,
  author       = {Pith},
  title        = {Pith review of: DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCFUBW4R}},
  note         = {Machine review of arXiv:2412.08069}
}

read the original abstract

Large Language Models (LLMs) have become increasingly integral to enhancing developer productivity, particularly in code generation, comprehension, and repair tasks. However, fine-tuning these models with high-quality, real-world data is challenging due to privacy concerns and the lack of accessible, labeled datasets. In this paper, we present DialogAgent, an automated tool for generating synthetic training data that closely mimics real developer interactions within Integrated Development Environments (IDEs). DialogAgent enables the production of diverse, high-fidelity query-response pairs by simulating multi-turn dialogues and contextual behaviors observed in real-world programming scenarios. The tool significantly reduces the reliance on manual data generation, increasing efficiency by 4.8 times compared to traditional methods. Our experiments and online deployment demonstrate substantial improvements in model performance for code-related question-answering tasks: the acceptance rate of responses generated by our in-house model is improved by 33%, after training on synthesized data generated by DialogAgent.

Figures

Figures reproduced from arXiv: 2412.08069 by the authors.

**Figure 2.** Process of QA-DBA [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Prompt for Behavior Judgment. ensures that the generated training data mirrors real-world developer interactions, maintaining consistency and diversity across different query types, code contexts, and development tasks. The LLM classifier and planner are both based on the Deepseek-Coder-V2-Instruct [24] model [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Prompt used by the query-generating. C. UI Automation in the IDE Plugin Due to the complexity of replicating real IDE environments and the lack of a low-cost solution to operate the IDE via APIs, we employ a UI automation tool to simulate user interactions within the I…

**Figure 7.** Figure 7: Framework of the Automation UI Tool [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Response Scoring Template 3 epochs and follows the experimental setup of [23]. During inference, we generated two response variations per prompt using a temperature of 0.3 and top-p of 0.95. The results were averaged over two rounds of inference. B. RQ1. Impact of High…

**Figure 9.** Figure 9: Response Comparison Template TABLE III: The Deduction Rules Scene Deduction Item Deduction Score Inline Chat Text description before the code 1 Chat View Lack of basic text description 1 Inline Chat & Chat View Language of response inconsistent with the instruction req…

**Figure 10.** Figure 10: Process of Scoring TABLE VIII: Manual Scoring Results for Different Generated Data Model PSR UR DS-33B-Inst-SFT(Ours) 46.64% 59.40% DS-33B-Inst-SFT(w/o MTD) 43.10% 52.86% DS-33B-Inst-SFT(subset) 46.28% 58.11% DS-33B-Inst-SFT(subset w/o MTD) 42.47% 52.18% DS-33B-Inst-S…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 7 canonical work pages

[1]

AnySphere, “Cursor,” https://www.cursor.com/, 2024

work page 2024
[2]

Github copilot,

“Github copilot,” https://github.com/features/copilot, accessed: 2024-05- 28

2024
[3]

Marscode,

“Marscode,” https://www.marscode.com/, accessed: 2024-05-28

2024
[4]

Codeium,

“Codeium,” https://codeium.com/, accessed: 2024-05-28

2024
[5]

Marscode agent: Ai- native automated bug fixing,

Y . Liu, P. Gao, X. Wang, C. Peng, and Z. Zhang, “Marscode agent: Ai- native automated bug fixing,” arXiv preprint arXiv:2409.00899 , 2024

arXiv 2024
[6]

Autocoderover: Autonomous program improvement,

Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604

2024
[7]

Agentless: De- mystifying llm-based software engineering agents,

C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024

arXiv 2024
[8]

Experiences from using code explanations generated by large language models in a web software development e-book,

S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen, “Experiences from using code explanations generated by large language models in a web software development e-book,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, 2023, pp. 931–937

2023

Show all 28 references

[9]

How abilities in large language models are affected by supervised fine-tuning data composition,

G. Dong, H. Yuan, K. Lu, C. Li, M. Xue, D. Liu, W. Wang, Z. Yuan, C. Zhou, and J. Zhou, “How abilities in large language models are affected by supervised fine-tuning data composition,” arXiv preprint arXiv:2310.05492, 2023

2023 arXiv
[10]

Learning to construct better mutation faults,

Z. Tian, J. Chen, Q. Zhu, J. Yang, and L. Zhang, “Learning to construct better mutation faults,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–13

2022
[11]

Codebert: A pre-trained model for programming and natural languages,

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

2002 arXiv
[12]

Codegen: An open large language model for code with multi-turn program synthesis,

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

2022 arXiv
[13]

Incoder: A generative model for code infilling and synthesis,

D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022

2022 arXiv
[14]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

2021 arXiv
[15]

Magicoder: Em- powering code generation with oss-instruct,

Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Em- powering code generation with oss-instruct,” in Forty-first International Conference on Machine Learning , 2024

2024
[16]

Mftcoder: Boosting code llms with multitask fine-tuning,

B. Liu, C. Chen, Z. Gong, C. Liao, H. Wang, Z. Lei, M. Liang, D. Chen, M. Shen, H. Zhou et al., “Mftcoder: Boosting code llms with multitask fine-tuning,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 5430–5441

2024
[17]

Large language models for data annotation: A survey,

Z. Tan, A. Beigi, S. Wang, R. Guo, A. Bhattacharjee, B. Jiang, M. Karami, J. Li, L. Cheng, and H. Liu, “Large language models for data annotation: A survey,” arXiv preprint arXiv:2402.13446 , 2024

2024 arXiv
[18]

Leveraging large language models (llms) to support collaborative human-ai online risk data annotation,

J. Park, P. Wisniewski, and V . Singh, “Leveraging large language models (llms) to support collaborative human-ai online risk data annotation,” arXiv preprint arXiv:2404.07926 , 2024

2024 arXiv
[19]

Enhancing human annotation: Leveraging large language models and efficient batch processing,

O. Zendel, J. S. Culpepper, F. Scholer, and P. Thomas, “Enhancing human annotation: Leveraging large language models and efficient batch processing,” in Proceedings of the 2024 Conference on Human Information Interaction and Retrieval , 2024, pp. 340–345

2024
[20]

Large language models as annotators: A preliminary evaluation for annotating low-resource language content,

S. Bhat and V . Varma, “Large language models as annotators: A preliminary evaluation for annotating low-resource language content,” in Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, 2023, pp. 100–107

2023
[21]

Fine-tuning large language models to improve accuracy and comprehensibility of automated code review,

Y . Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y . Xu, and J. Wang, “Fine-tuning large language models to improve accuracy and comprehensibility of automated code review,” ACM Transactions on Software Engineering and Methodology , 2024

2024
[22]

Training language models to follow instructions with human feedback,

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

2022
[23]

Deepseek-coder: When the large language model meets programming -the rise of code intelligence,

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming -the rise of code intelligence,” arXiv preprint arXiv:2401.14196 , 2024

2024 arXiv
[24]

Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Ma, W. Zeng et al. , “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931, 2024

2024 arXiv
[25]

Starcoder 2 and the stack v2: The next generation,

A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii,...

2024
[26]

Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model,

DeepSeek-AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin et al., “Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model,” arXiv preprint arXiv:2405.04434 , 2024

2024 arXiv
[27]

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[28]

Judging llm-as-a-judge with mt-bench and chatbot arena,

L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” arXiv preprint arXiv:2306.05685 , 2023

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

AnySphere, “Cursor,” https://www.cursor.com/, 2024

work page 2024

[2] [2]

Github copilot,

“Github copilot,” https://github.com/features/copilot, accessed: 2024-05- 28

2024

[3] [3]

Marscode,

“Marscode,” https://www.marscode.com/, accessed: 2024-05-28

2024

[4] [4]

Codeium,

“Codeium,” https://codeium.com/, accessed: 2024-05-28

2024

[5] [5]

Marscode agent: Ai- native automated bug fixing,

Y . Liu, P. Gao, X. Wang, C. Peng, and Z. Zhang, “Marscode agent: Ai- native automated bug fixing,” arXiv preprint arXiv:2409.00899 , 2024

arXiv 2024

[6] [6]

Autocoderover: Autonomous program improvement,

Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604

2024

[7] [7]

Agentless: De- mystifying llm-based software engineering agents,

C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024

arXiv 2024

[8] [8]

Experiences from using code explanations generated by large language models in a web software development e-book,

S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen, “Experiences from using code explanations generated by large language models in a web software development e-book,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, 2023, pp. 931–937

2023