Cloning a Conversational Voice AI Agent from Call\,Recording Datasets for Telesales

Krittanon Kaewtawee; Krittin Pachtrachai; Touchapon Kraisingkorn; Wachiravit Modecrua

REVIEW 4 major objections 5 minor 24 references

Cloning a Conversational Voice AI Agent from Call\,Recording Datasets for Telesales

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a telesales voice agent can be cloned from call recordings by distilling top agents' behaviour into a system prompt, with blind tests showing parity on routine segments and a gap on persuasion.

desk verdict A practical voice-agent cloning pipeline with a real but underspecified evaluation; the 'approaches human performance' claim outruns the data. read the letter →

arxiv 2509.04871 v1 pith:QSINNW7H submitted 2025-09-05 cs.AI cs.LG

Krittanon Kaewtawee , Wachiravit Modecrua , Krittin Pachtrachai , Touchapon Kraisingkorn This is my paper · ORCID

classification cs.AIcs.LG

keywords voiceAIagentsagentcloningpromptengineeringtelesaleslargelanguagemodelsspeech-to-speechblindevaluationcallcentreautomation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a working conversational voice agent for telesales can be cloned from recorded calls without training a new model: top-performing calls are mined for their structure and tactics, and that knowledge is loaded into a general large language model as a structured system prompt (the 'Agent Playbook'). In blind tests scored by seven reviewers on a 22-criterion rubric, the cloned agent performs on par with human agents on introduction and product communication, while underperforming on sales drive and objection handling; a second version, refined after error analysis and fine-tuned on roughly sixty examples, closes much of that gap. If the claim holds, any organization that records its customer calls could build a domain-specific voice agent from transcripts alone and at a fraction of the cost of training a model. The paper's stated conclusion is a hybrid workforce: AI carries the routine parts of calls, human agents keep the persuasion and complex objections.

What carries the argument

The load-bearing object is the Agent Playbook: a single system prompt that bundles a job description, a knowledge manual, example dialogues, conversation-stage guidelines, objection-handling tactics, compliance rules, and customer-context slots into instructions a general LLM can follow. It carries the argument because it is the channel through which skill transfers from recorded human calls to the model—the clone is not trained from scratch, and everything the agent knows enters through this prompt plus a small fine-tuning set. Around it sit the cloning pipeline that builds the playbook from sampled, ranked recordings, and the runtime stack—a streaming speech-to-speech API over WebSockets—t

What would settle it

Deploy the cloned agent on calls the playbook was not built from: a live outbound campaign or a held-out set of scripted scenarios whose objections and customer behaviours were excluded from curation, and compare installation-booking or appointment rates against human agents on the same leads. The 'approaches human performance' claim would be settled by whether the gap seen in the three-scenario study widens on unseen objections, or by whether independent raters agree on the top-vs-average ranking that feeds the playbook.

Watch

Extended reading notes

Core claim

The paper's central claim is that a human-like telesales voice agent can be produced from call recordings by extracting what top agents do well and encoding it as a prompt. The cloning pipeline samples about 1,000 recorded calls, ranks them by agent quality, studies roughly 40 high-performing calls to draft the agent's job description, extracts product knowledge, common objections, persuasive techniques, and closing strategies into a knowledge manual, distills representative dialogues, and composes everything into a single system prompt—the 'Agent Playbook'—that is handed to an instruction-tuned large language model. At runtime the agent runs on a speech-to-speech streaming API, taking audio

Load-bearing premise

The whole result rests on the evaluation being a fair measure of real telesales skill: three scripted scenarios, seven blind reviewers, a rubric built with one company's trainers, and a top-vs-average ranking of about 1,000 calls whose criteria are never stated—if any of these is unrepresentative, the claim that the agent approaches human performance is unsupported.

Editorial extensions

If this is right

Any organization with a corpus of call recordings could build a working domain voice agent without training a model, provided it can rank calls by quality and curate strong examples.
Prompt quality is a measurable performance variable: fixing an ambiguous success criterion, trimming redundancy, removing list formatting, and softening excessive politeness raised objection handling and sales drive scores by roughly 20%.
The results support the paper's hybrid-staffing conclusion: AI can be trusted with routine call segments, while persuasion and complex objection handling remain human strengths.
The playbook doubles as a written training manual for human agents, since it encodes the same persona, facts, tactics, and compliance rules the human team operates under.
The authors claim the same cloning pipeline generalizes beyond sales to any domain with call transcripts, such as customer service or healthcare.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the playbook is built from calls the authors ranked as 'top quality' without specifying the ranking criteria, the whole clone inherits whatever bias that ranking has; making the ranking explicit and measuring inter-rater agreement would be a cheap, direct robustness check of the method.
Editorial inference: the same prompt-composition pipeline could be run in reverse as an auditing or coaching tool—scoring real human agents against the 22-criterion rubric to find which playbook sections their calls deviate from.
Editorial inference: the most decisive test the paper does not run is an outcome-based one—comparing installation-booking rates of the AI agent and human agents on live outbound campaigns, since a small rubric gap may be commercially irrelevant while a small rubric win may not survive contact with unscripted customers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper's actual contribution is a concrete, working pipeline: take ~1,000 call recordings, rank them, extract a job description, knowledge manual, and example dialogues, and compile them into a system prompt that drives a speech-to-speech model. That is a useful engineering recipe, and the authors are honest that the agent lags humans on objection handling and sales drive. The blind evaluation with 22 criteria and seven evaluators is a reasonable first check, and the error analysis and prompt refinement section is the best part—specific failure modes tied to prompt details, with a plausible mechanism for each fix.

The soft spots are in the comparison. The central claim is that the AI 'approaches human performance,' but the experiment uses one human per scenario. No replication across human agents, so the human benchmark is a single point per scenario, and error bars capture only evaluator disagreement, not human variance. That single human could be atypical, especially in the hard scenarios, so the comparative conclusion is not established. Also the ranking step—classifying ~1,000 calls into top vs. average—is never specified. The criteria are left to the imagination, and the selection of the 40 high-performing calls is unvalidated. That matters because it is the foundation of the playbook.

The generalization claim is overbroad. The method is tied to telesales structure; saying it generalizes to any domain with transcripts ignores that telesales has a predictable opening-pitch-objection-close shape that many domains lack. The authors do acknowledge limits in the conclusion, but the abstract overreaches.

The paper is not a scientific breakthrough, but it is a serious engineering contribution with a reproducible-ish pipeline (if they ever release code and prompts). As a preprint it is missing the artifacts needed for independent verification, and the evaluation design should be tightened before publishing. That said, it deserves a referee: the method is clearly described, the failure analysis is informative, and a competent reviewer could push the authors to fix the comparison and the ranking documentation.

If you have a colleague working on applied voice agents or call center automation, point them to this. The evaluation is a useful template for the field, just not the final word.

Referee Report

4 major / 5 minor

Summary. The paper presents a methodology for cloning a telesales voice AI agent from call recordings. The cloning pipeline samples and ranks calls, extracts a job description, product knowledge, example dialogues, and compliance rules, then composes them into a system prompt for a Gemini Live API-based inference system. The agent is evaluated against human agents on three scripted scenarios using a 22-criteria rubric scored by seven blind evaluators. The central claim is that the AI approaches human performance on routine call segments (introduction, product communication) but underperforms in persuasion and objection handling; prompt refinement and light fine-tuning improve the latter scores. The paper concludes with lessons and future work involving simulation and automated evaluation.

Significance. If the result holds, the paper offers a practical, low-cost template for building domain-specific voice agents from call corpora, with a clear separation between knowledge extraction and runtime inference. Strengths include a rubric co-developed with the company's sales trainers, blind scoring by evaluators not involved in system development, and an explicit error-analysis/refinement loop. The work is engineering-oriented and reproducible in spirit, though no code or data are released. The main limitation is the evaluation design: with three scenarios, seven evaluators, and one human agent per scenario, the comparative claim 'approaches human performance' is not statistically grounded. The ranking step in Section 2 also lacks specification and validation. These issues are addressable but require additional experimentation.

major comments (4)

[Section 4, Figure 3] The central comparative claim rests on one human agent per scenario. Each scenario uses a single paired call (one human, one AI); the seven evaluators score the same recordings, so the reported error bars capture evaluator disagreement only, not agent-to-agent variance. There is no confidence interval for the human benchmark, no inter-agent replication, and no inferential statistical test. The data support 'this AI scored comparably to one particular human on three scripted calls,' not 'approaches human performance' as a general statement. Please add multiple human agents per scenario, report agent-level variance, and use appropriate statistical modeling (e.g., mixed-effects with evaluator random effects).
[Section 2, 'Sampling and ranking'] The cloning pipeline begins by sampling roughly 1,000 calls and classifying them into 'top vs. average' quality, then selecting about 40 high-performing calls for detailed analysis. The criteria for this ranking are never specified, and no inter-rater reliability or validation is reported. Because the entire playbook is derived from this curated subset, any bias in the ranking propagates directly into the agent's behavior. Please define the ranking protocol, who performed it, and how the 40-call sample was confirmed to contain the full range of conversational patterns needed for the agent.
[Section 4, 'Test scenarios'] Only three scripted scenarios are evaluated: happy path, negotiation, and complaining customer. These were chosen by the authors, but no evidence is provided that they are representative of the distribution of call types in the ~1,000-call corpus. The abstract's generalization to 'routine aspects of the call' requires that the test scenarios resemble routine telesales calls. Please justify the scenario selection using corpus statistics, or evaluate on a larger, more diverse set of scenarios sampled from the data.
[Section 5, Figure 4] The V2 results are reported on the same three scenarios that motivated the prompt refinement. Although the evaluation uses external evaluators and comments (not direct score fitting), iterating on the same test set risks overfitting to these specific scripts. The 'roughly 20%' improvement and 'significant gains' claims lack statistical support and a held-out evaluation set. Please provide a separate validation set or a sensitivity analysis to demonstrate that the refinements generalize beyond the three scenarios.

minor comments (5)

[Introduction] Typo: 'healthcare care' should be 'healthcare' or 'health care.' Also, 'we describe detail the architectural design' should be 'we describe in detail.'
[Figure 1] The text 'High Quality Recoding' should be 'High Quality Recording.'
[Section 3, item 5] The expansion of RAG (retrieval-augmented generation) is out of place in product knowledge for a telesales agent; if RAG is not used in the system, remove this mention.
[Title page] The footnote '†Reviewer' attached to an author's name is confusing and should be corrected; the symbol likely denotes a reviewing role, not an affiliation.
[Section 4 and 5] Phrases such as 'on par with human agents' and 'significant gains' are not backed by statistical tests; consider softening or adding error bars/confidence intervals that reflect the actual design.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cloning pipeline is an inductive extraction from call recordings, and the human benchmark is an external, blind evaluation independent of the model's outputs.

full rationale

The paper's derivation chain is self-contained and not circular. The cloning pipeline (Section 2) transforms call recordings into a structured system prompt via job-description drafting, knowledge extraction, example dialogue generation, and prompt composition; this is an inductive process from the data, not a definitional restatement of the evaluation targets. The evaluation (Section 4) uses a 22-criterion rubric developed with the company's sales trainers, three scripted scenarios, and blind scoring by seven evaluators, so the human-agent comparison is an external benchmark independent of the model's outputs. Prompt refinement (Section 5) is iterative development based on evaluator comments and failure-mode analysis, not fitting a parameter to the evaluation target; no scoring rule or evaluation outcome is used as an input to the prompt construction. There are no load-bearing self-citations: the cited prior work (e.g., references [7], [11], [23]) is not authored by the present paper's authors and is used for context rather than to justify the cloning claim. The unspecified ranking step in Section 2 and the single-human-agent-per-scenario comparison are measurement/validity concerns, not circularity: they do not make the claimed result equivalent to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted numerical parameters. Its epistemic weight rests entirely on domain assumptions about the sufficiency of prompt engineering, the validity of the rubric and the ranking process, and the representativeness of the small evaluation set. These assumptions are plausible but unvalidated, and they are load-bearing for the generalization claim.

assumptions (4)

domain assumption Prompt engineering can shape a general LLM to emulate top-performing human agents without extensive training.
The entire cloning method rests on this premise. It is invoked in the abstract and throughout Section 3, where the system prompt is claimed to encode the agent's persona and best practices from call recordings.
domain assumption The 22-criteria rubric developed with the company's sales trainers captures what constitutes an excellent sales call.
Used as the ground truth for evaluation in Section 4. No validation of the rubric's reliability or external validity is provided.
domain assumption Sampling roughly 1,000 calls and classifying them into 'top' vs 'average' quality is a valid way to identify high-quality interactions.
Described in the first bullet of Section 2, but the ranking criteria, inter-rater reliability, and selection procedure are never specified, yet this step determines the knowledge that goes into the playbook.
domain assumption Blind scoring by seven evaluators on three scripted scenarios is sufficient to measure agent competence and support claims of 'approaching human performance'.
The entire evaluation in Section 4 relies on this assumption. The small sample and absence of statistical power analysis are not discussed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cloning a Conversational Voice AI Agent from Call\,Recording Datasets for Telesales." pith.science (2026). https://pith.science/paper/QSINNW7H

@misc{pith2026250904871,
  author       = {Pith},
  title        = {Pith review of: Cloning a Conversational Voice AI Agent from Call\,Recording Datasets for Telesales},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QSINNW7H}},
  note         = {Machine review of arXiv:2509.04871}
}

read the original abstract

Recent advances in language and speech modelling have made it possible to build autonomous voice assistants that understand and generate human dialogue in real time. These systems are increasingly being deployed in domains such as customer service and healthcare care, where they can automate repetitive tasks, reduce operational costs, and provide constant support around the clock. In this paper, we present a general methodology for cloning a conversational voice AI agent from a corpus of call recordings. Although the case study described in this paper uses telesales data to illustrate the approach, the underlying process generalizes to any domain where call transcripts are available. Our system listens to customers over the telephone, responds with a synthetic voice, and follows a structured playbook learned from top performing human agents. We describe the domain selection, knowledge extraction, and prompt engineering used to construct the agent, integrating automatic speech recognition, a large language model based dialogue manager, and text to speech synthesis into a streaming inference pipeline. The cloned agent is evaluated against human agents on a rubric of 22 criteria covering introduction, product communication, sales drive, objection handling, and closing. Blind tests show that the AI agent approaches human performance in routine aspects of the call while underperforming in persuasion and objection handling. We analyze these shortcomings and refine the prompt accordingly. The paper concludes with design lessons and avenues for future research, including large scale simulation and automated evaluation.

Figures

Figures reproduced from arXiv: 2509.04871 by the authors.

**Figure 2.** The example to show the input / output of cloning engine that extract the factual [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Initial evaluation results comparing the AI agent to human agents. Scores are averaged [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Evaluation results after prompt optimisation and fine-tuning (AI agent V2). The AI’s [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

[1]

Language models are few-shot learners,

T. B. Brown, B. Mann, N. Ryder,et al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems 33, 2020

work page 2020
[2]

Training language models to follow instructions with human feedback,

L. Ouyang, J. Wu, X. Jiang,et al., “Training language models to follow instructions with human feedback,” inAdvances in Neural Information Processing Systems, 2022

work page 2022
[3]

Slm: Bridge the thin gap between speech and text foundation models,

M. Wang, W. Han, I. Shafran, Z. Wu, J. Schalkwyk,et al., “Slm: Bridge the thin gap between speech and text foundation models,”arXiv preprint arXiv:2310.00230, 2023. preprint

arXiv 2023
[4]

Towards end-to-end spoken language understanding,

D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,”arXiv preprint arXiv:1802.08395, 2018. preprint

arXiv 2018
[5]

Amazon nova sonic: Technical report and model card,

A. A. G. I. Team, “Amazon nova sonic: Technical report and model card,” tech. rep., Ama- zon Scientific Research, 2025. Technical report and model card describing the design and capabilities of the Amazon Nova Sonic speech-to-speech foundation model

work page 2025
[6]

Neural codec language models are zero-shot text to speech synthesizers,

C. Wang, S. Chen, Y. Wu,et al., “Neural codec language models are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

arXiv 2023
[7]

Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS

V. Ethiraj, A. David, S. Menon, and D. Vijay, “Toward low-latency end-to-end voice agents for telecommunications using streaming asr, quantized llms and real-time tts,”arXiv preprint arXiv:2508.04721, 2025

work page Pith review arXiv 2025
[8]

Microsoft saved over$500 million by using ai in its call centers,

I. P. staff, “Microsoft saved over$500 million by using ai in its call centers,”ITPro, 2025. Chief Commercial Officer Judson Althoff confirmed that in 2024, Microsoft achieved over$500 million in savings, primarily from its call centers

work page 2025

Show all 24 references

[9]

Voice ai agents market size, share, trends analysis 2024–2034,

Market.us Research, “Voice ai agents market size, share, trends analysis 2024–2034,” tech. rep., apr 2025. 9

2024
[10]

Efficient customer service combining human operators and virtual agents,

Y. Oshrat, Y. Aumann, T. Hollander, O. Maksimov, A. Ostroumov, N. Shechtman, and S. Kraus, “Efficient customer service combining human operators and virtual agents,”arXiv preprint arXiv:2209.05226, 2022

2022 arXiv
[11]

Customer service representative’s perception of the ai assistant in an organisation’s call center,

K. Qin, K. Du, Y. Chen,et al., “Customer service representative’s perception of the ai assistant in an organisation’s call center,”arXiv preprint arXiv:2507.00513, 2025

2025
[12]

Contact center ai assistants are introducing new inefficiencies and burdens, finds study,

C. Today, “Contact center ai assistants are introducing new inefficiencies and burdens, finds study,” Jul 2025

2025
[13]

The mental health implications of artificial intelligence adoption in the workplace,

B. J. Kim, “The mental health implications of artificial intelligence adoption in the workplace,” Nature Human Behaviour, vol. 8, no. 4, pp. 423–431, 2024

2024
[14]

How artificial intelligence-induced job insecurity shapes psychological safety and knowledge-hiding behavior,

B. J. Kim, “How artificial intelligence-induced job insecurity shapes psychological safety and knowledge-hiding behavior,”Journal of Business Research, vol. 144, pp. 123–132, 2024

2024
[15]

Ai and employee wellbeing in the workplace: An empirical study,

A. Valtonen, “Ai and employee wellbeing in the workplace: An empirical study,”Journal of Business Research, vol. 148, pp. 456–465, 2025

2025
[16]

The contact center crossroads: Finding the right mix of humans and ai,

M. . Company, “The contact center crossroads: Finding the right mix of humans and ai,” 2025

2025
[17]

Generative ai can boost productivity without replacing workers,

E. Brynjolfsson, D. Li, and L. R. Raymond, “Generative ai can boost productivity without replacing workers,”Stanford Graduate School of Business, 2023. Accessed: 2025-09-01

2023
[18]

Comparing ai and human decision-making mechanisms in daily tasks,

L. Wang, “Comparing ai and human decision-making mechanisms in daily tasks,”PMC, 2025. Accessed: 2025-09-01

2025
[19]

Why the future of customer service depends on human-ai collaboration,

CMSWire, “Why the future of customer service depends on human-ai collaboration,”CM- SWire, 2025. Accessed: 2025-09-01

2025
[20]

Voice-based ai in call center cus- tomer service: A natural field experiment,

L. Wang, N. Huang, Y. Hong, L. Liu, X. Guo, and G. Chen, “Voice-based ai in call center cus- tomer service: A natural field experiment,”Production and Operations Management, vol. 32, no. 4, pp. 1002–1018, 2023

2023
[21]

Realtime api guide

OpenAI, “Realtime api guide.”https://platform.openai.com/docs/guides/realtime,
[22]

Ultravox: A fast multimodal llm for real-time voice interactions,

Fixie.ai, “Ultravox: A fast multimodal llm for real-time voice interactions,” 2025. Available athttps://github.com/fixie-ai/ultravox

2025
[23]

Future of work with ai agents: Auditing automation and augmentation potential across the u.s. workforce,

Y. J. Shao, H. Zope, Y. Jiang,et al., “Future of work with ai agents: Auditing automation and augmentation potential across the u.s. workforce,”arXiv preprint arXiv:2506.06576, 2025. 10

2025
[2025]

Accessed: 2025-09-01

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Language models are few-shot learners,

T. B. Brown, B. Mann, N. Ryder,et al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems 33, 2020

work page 2020

[2] [2]

Training language models to follow instructions with human feedback,

L. Ouyang, J. Wu, X. Jiang,et al., “Training language models to follow instructions with human feedback,” inAdvances in Neural Information Processing Systems, 2022

work page 2022

[3] [3]

Slm: Bridge the thin gap between speech and text foundation models,

M. Wang, W. Han, I. Shafran, Z. Wu, J. Schalkwyk,et al., “Slm: Bridge the thin gap between speech and text foundation models,”arXiv preprint arXiv:2310.00230, 2023. preprint

arXiv 2023

[4] [4]

Towards end-to-end spoken language understanding,

D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,”arXiv preprint arXiv:1802.08395, 2018. preprint

arXiv 2018

[5] [5]

Amazon nova sonic: Technical report and model card,

A. A. G. I. Team, “Amazon nova sonic: Technical report and model card,” tech. rep., Ama- zon Scientific Research, 2025. Technical report and model card describing the design and capabilities of the Amazon Nova Sonic speech-to-speech foundation model

work page 2025

[6] [6]

Neural codec language models are zero-shot text to speech synthesizers,

C. Wang, S. Chen, Y. Wu,et al., “Neural codec language models are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

arXiv 2023

[7] [7]

Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS

V. Ethiraj, A. David, S. Menon, and D. Vijay, “Toward low-latency end-to-end voice agents for telecommunications using streaming asr, quantized llms and real-time tts,”arXiv preprint arXiv:2508.04721, 2025

work page Pith review arXiv 2025

[8] [8]

Microsoft saved over$500 million by using ai in its call centers,

I. P. staff, “Microsoft saved over$500 million by using ai in its call centers,”ITPro, 2025. Chief Commercial Officer Judson Althoff confirmed that in 2024, Microsoft achieved over$500 million in savings, primarily from its call centers

work page 2025