The Impact of AI Tool on Engineering at ANZ Bank An Empirical Study on GitHub Copilot within Corporate Environment

Ching Louis Liu; Gareth Rowland; Sayan Chatterjee; Tim Hogarth

RECORD

The Impact of AI Tool on Engineering at ANZ Bank An Empirical Study on GitHub Copilot within Corporate Environment

Sayan Chatterjee, Ching Louis Liu, Gareth Rowland, Tim Hogarth

Software Engineering · Submitted 8 Feb 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Results showed a notable boost in productivity and code quality with GitHub Copilot, though its impact on code security remained inconclusive.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2402.05636 (version 2)

Pith ID5CCIVH5F, Pith's permanent record of this paper and its review

CitationChing Louis Liu, Gareth Rowland, Sayan Chatterjee, Tim Hogarth. "The Impact of AI Tool on Engineering at ANZ Bank An Empirical Study on GitHub Copilot within Corporate Environment." arXiv:2402.05636 (2024). https://pith.science/paper/5CCIVH5F

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,725 input tokens.

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.77 · human participants 0.98 · controlled comparison 0.98 · randomized assignment 0.05 · states quantitative result 0.84 · compares with prior results 0.67 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

The increasing popularity of AI, particularly Large Language Models (LLMs), has significantly impacted various domains, including Software Engineering. This study explores the integration of AI tools in software engineering practices within a large organization. We focus on ANZ Bank, which employs over 5000 engineers covering all aspects of the software development life cycle. This paper details an experiment conducted using GitHub Copilot, a notable AI tool, within a controlled environment to evaluate its effectiveness in real-world engineering tasks. Additionally, this paper shares initial findings on the productivity improvements observed after GitHub Copilot was adopted on a large scale, with about 1000 engineers using it. ANZ Bank's six-week experiment with GitHub Copilot included two weeks of preparation and four weeks of active testing. The study evaluated participant sentiment and the tool's impact on productivity, code quality, and security. Initially, participants used GitHub Copilot for proposed use-cases, with their feedback gathered through regular surveys. In the second phase, they were divided into Control and Copilot groups, each tackling the same Python challenges, and their experiences were again surveyed. Results showed a notable boost in productivity and code quality with GitHub Copilot, though its impact on code security remained inconclusive. Participant responses were overall positive, confirming GitHub Copilot's effectiveness in large-scale software engineering environments. Early data from 1000 engineers also indicated a significant increase in productivity and job satisfaction.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	The increasing popularity of AI, particularly Large Language Models (LLMs), has significantly impacted various domains, including Software Engineering.	background 1.00	0.04	0.02	asserted 1.00
s1	This study explores the integration of AI tools in software engineering practices within a large organization.	contribution 0.95	0.49	0.02	asserted 0.70
s2	We focus on ANZ Bank, which employs over 5000 engineers covering all aspects of the software development life cycle.	background 0.58	0.20	0.02	asserted 1.00
s3	This paper details an experiment conducted using GitHub Copilot, a notable AI tool, within a controlled environment to evaluate its effectiveness in real-world engineering tasks.	method 0.50	0.90	0.03	asserted 0.95
s4	Additionally, this paper shares initial findings on the productivity improvements observed after GitHub Copilot was adopted on a large scale, with about 1000 engineers using it.	result 0.67	0.87	0.11	asserted 0.80
s5	ANZ Bank's six-week experiment with GitHub Copilot included two weeks of preparation and four weeks of active testing.	method 0.99	0.88	0.02	asserted 1.00
s6	The study evaluated participant sentiment and the tool's impact on productivity, code quality, and security.	method 0.80	0.91	0.04	asserted 0.94
s7	Initially, participants used GitHub Copilot for proposed use-cases, with their feedback gathered through regular surveys.	method 1.00	0.90	0.02	asserted 1.00
s8	In the second phase, they were divided into Control and Copilot groups, each tackling the same Python challenges, and their experiences were again surveyed.	method 1.00	0.94	0.02	asserted 0.99
s9	Results showed a notable boost in productivity and code quality with GitHub Copilot, though its impact on code security remained inconclusive.	result 1.00	0.95	0.65	asserted 1.00
s10	Participant responses were overall positive, confirming GitHub Copilot's effectiveness in large-scale software engineering environments.	interpretation 0.52	0.89	0.43	asserted 1.00
s11	Early data from 1000 engineers also indicated a significant increase in productivity and job satisfaction.	result 1.00	0.91	0.15	hedged possibility 0.88

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

The increasing popularity of AI, particularly Large Language Models (LLMs), has significantly impacted various domains, including Software Engineering. This study explores the integration of AI tools in software engineering practices within a large organization. We focus on ANZ Bank, which employs over 5000 engineers covering all aspects of the software development life cycle. This paper details an experiment conducted using GitHub Copilot, a notable AI tool, within a controlled environment to evaluate its effectiveness in real-world engineering tasks. Additionally, this paper shares initial findings on the productivity improvements observed after GitHub Copilot was adopted on a large scale, with about 1000 engineers using it. ANZ Bank's six-week experiment with GitHub Copilot included two weeks of preparation and four weeks of active testing. The study evaluated participant sentiment and the tool's impact on productivity, code quality, and security. Initially, participants used GitHub Copilot for proposed use-cases, with their feedback gathered through regular surveys. In the second phase, they were divided into Control and Copilot groups, each tackling the same Python challenges, and their experiences were again surveyed. Results showed a notable boost in productivity and code quality with GitHub Copilot, though its impact on code security remained inconclusive. Participant responses were overall positive, confirming GitHub Copilot's effectiveness in large-scale software engineering environments. Early data from 1000 engineers also indicated a significant increase in productivity and job satisfaction.

Human review

How Pith checks papers → · Full audit of this paper →

References

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

On the Prospects of Dynamic LLM Conversations in Software Development
cs.SE 2026-08 conditional novelty 5.0 of 10

Minimal context and proactive guidance in LLM chats did not harm measured developer behavior and showed trends toward higher satisfaction with proactive interventions.
Adoption of Generative Artificial Intelligence in the German Software Engineering Industry: An Empirical Study
cs.SE 2026-01 conditional novelty 4.0 of 10

In a survey of 109 German developers plus 18 interviews, GenAI productivity gains cluster among 'power users'; junior and senior developers perceive prompting differently, and limited codebase context is a major barrier.
Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review
cs.CL 2025-05 conditional novelty 4.0 of 10

A systematic review of 54 studies finds that generative language modelling for low-resource languages relies mostly on transformer models, covers only a small set of languages, and lacks consistent evaluation.
Experience with GitHub Copilot for Developer Productivity at Zoominfo
cs.SE 2025-01 conditional novelty 4.0 of 10

At ZoomInfo, GitHub Copilot suggestions were accepted 33% of the time with 20% of lines accepted, alongside 72% developer satisfaction, in line with earlier vendor and industry reports.

How the literature uses it

6 arXiv papers cite this in 10 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 7 sentences without a confident label

Cites it as support 2 papers

Field studies in industry also found increases in number of completed tasks, with less experienced developers showing greater gains [ref] .

The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study · Productivity and Workflow · 0.96
Given that language models could be productivity-enhancing [ref] , their incompatibility with entire language families could introduce a new dimension of economic inequality.

Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review · RQ1: What Low-Resource Languages Have Already Been Modelled by Other Studies? · 0.97

Cites it as background 1 paper

Recent work has explored the potential of these tools to assist students and novice programmers [ref] , as well as programmers' perceptions of [ref] and trust in [ref] AI tools.

How much does AI impact development speed? An enterprise-based randomized controlled trial · Related Work · 1.00

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.