Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems

Ali Ansari; Amirhossein Afsharrad; Aruj Mahajan; Nima Yazdani; Rumi Allbert; Seyed Shahabeddin Mousavi

REVIEW 4 major objections 5 minor 9 references

Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a Google STT + GPT-4.1 + Cartesia TTS stack outperforms four other production configurations for AI-conducted interviews on both automated quality metrics and user ratings.

desk verdict A useful, large-scale production comparison of STT×LLM×TTS stacks, but the headline ranking is plausible rather than established because deployments were non-overlapping and the LLM judge is not validated in-paper. read the letter →

arxiv 2507.16835 v2 pith:FUBBGDNQ submitted 2025-07-15 eess.AS cs.CL

Rumi Allbert , Nima Yazdani , Ali Ansari , Aruj Mahajan , Amirhossein Afsharrad , Seyed Shahabeddin Mousavi This is my paper · ORCID

classification eess.AScs.CL

keywords speech-to-texttext-to-speechlargelanguagemodelsLLM-as-a-JudgeconversationalAIcascadedarchitectureusersatisfactioninterviews

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which combinations of speech-to-text, large language model, and text-to-speech components work best in a voice-based AI interviewer. Analyzing transcripts from thousands of real job interviews, it claims that a stack with Google's STT, GPT-4.1, and Cartesia's TTS beats four other production configurations on LLM-judged conversational and technical quality and on candidate ratings. It also claims that these objective quality metrics correlate only weakly with user satisfaction, so automated scores do not capture what makes candidates happy. A sympathetic reader would take the paper's three contributions as a production-scale comparison, a validated LLM-judge evaluation method, and evidence that component choice in cascaded voice AI matters most at the transcription stage.

What carries the argument

The argument is carried by a production system whose STT, LLM, and TTS modules can be swapped independently, creating five naturally occurring configurations; by LLM-as-a-Judge evaluation, in which a Claude 3.5 Sonnet model scores each transcript on conversational sub-metrics (dialogue flow, response building, acknowledgement) and technical sub-metrics (skill alignment, logical progression, question clarity); and by statistical machinery of Levene's tests, Welch's ANOVA, Games-Howell post-hoc tests, and Pearson correlations that turn score distributions into comparisons. The same five configurations are re-sliced by component to attribute differences to STT, LLM, or TTS.

What would settle it

Re-run the five configurations in randomized, overlapping deployment windows, or add interview date as a covariate to the Welch ANOVA; if the advantage of Google STT + GPT-4.1 + Cartesia TTS disappears or shrinks to non-significance, the reported ordering is an artifact of timing rather than component quality.

Watch

Extended reading notes

Core claim

The central claim is that component choice in a cascaded STT x LLM x TTS pipeline materially changes both measured interview quality and user satisfaction. Using five production configurations on over 5,000 interviews drawn from a system running about 1,500 interviews per day, the paper reports that Google STT + GPT-4.1 + Cartesia TTS scored highest on accuracy (8.12), conversational quality (8.78), technical quality (8.57), and average user rating (4.53), with statistically significant Welch ANOVA differences for every main metric (soft skills excluded because n=10). A second claim is that automated LLM-judge metrics and candidate star ratings are nearly orthogonal, with most correlations below 0.11, which the authors interpret as evidence that user experience depends on factors like perceived empathy and voice naturalness that technical quality scores miss. A third claim is that decomposition of the five configurations shows STT has the dominant effect, the LLM a moderate effect, and TTS a smaller but systematic effect.

Load-bearing premise

The comparison assumes that the five configurations were tested under comparable conditions; because deployments did not overlap in time, seasonal candidate differences or unrelated system improvements could produce the same pattern.

Editorial extensions

If this is right

Engineers building cascaded voice AI systems should prioritize STT quality, since the paper attributes the largest performance differences to the transcription component.
The Google STT + GPT-4.1 + Cartesia TTS combination is the current best-performing production stack for AI-conducted interviews.
Objective LLM-judge scores should be supplemented with direct user feedback, because the paper finds these metrics capture largely different aspects of system performance.
LLM-as-a-Judge can serve as a scalable, cost-effective evaluation method for voice-based conversational AI, though its scores will not fully predict user satisfaction.
Improving the TTS component yields smaller but consistent gains, which may still matter for user experience in deployed voice systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper, the weak correlation between automated quality and user satisfaction suggests that optimizing purely for objective conversational quality may not raise user ratings; factors such as perceived empathy, voice naturalness, and interaction design may need independent measurement.
The five configurations are not a full factorial design, so a fully crossed experiment with multiple STT, LLM, and TTS options would disentangle interaction effects that the current component decomposition can only approximate.
If transcription errors indeed cascade, then testing robustness under noisy audio or accented speech could show even larger STT effects than the average-case comparison reported here.
Using an ensemble of multiple LLM judges, as the paper names for future work, might produce scores that correlate more strongly with user satisfaction than a single judge's scores.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful, readable production study of five STT×LLM×TTS stacks in a real AI-interview system, built on a big dataset (over 5,000 sampled interviews from 300,000+). The descriptive ranking and the weak correlation between LLM-judge scores and user satisfaction are worth knowing. But the headline causal claim — that Google STT + GPT-4.1 + Cartesia is the best stack — is undercut by non-overlapping deployment periods, which the authors themselves acknowledge in §5.1. Treat this as a plausible ranking, not an established result.

What is genuinely good: the modular swap design yields pairwise controlled comparisons (C1 vs C3 isolates TTS, C2 vs C4 isolates STT), the statistical analysis is appropriate (Levene, Welch ANOVA, Games-Howell), and the appendix contains the full judge prompt. The weak correlation observation (most correlations with user ratings below 0.11) is a genuinely interesting caution for anyone building automated evaluation pipelines. The paper also flags its own limitations: self-selected ratings, domain specificity, limited TTS variety.

The soft spots, in proportion: the temporal confound is the big one. Configurations were deployed sequentially, not concurrently or randomly, so C3's higher means could reflect candidate season, pipeline improvements, or candidate pool shifts rather than component quality. The abstract overstates the user-satisfaction claim: the post-hoc test shows C3 was not significantly better than C1 on rate star, only better than C2, C4, and C5. The LLM judge is validated by citing the authors' prior paper [1], with no agreement statistics reported here. No data release, and per-configuration sample sizes are missing, which weakens effect-size interpretation. The component contribution analysis is not a full factorial, though the pairwise contrasts help.

Who this is for: product teams choosing voice AI components, and researchers interested in LLM-as-a-judge limits. It deserves serious referee time — the scale and honest framing outweigh the confound — but it needs revision: soften the causal language, add concurrent or controlled deployment evidence, and validate the judge in-paper. I'd send it to review.

Referee Report

4 major / 5 minor

Summary. The paper reports a production-scale comparison of five STT x LLM x TTS configurations used in an AI-conducted job interview system. Using LLM-as-a-Judge scores (Claude 3.5 Sonnet) and post-interview user ratings, it reports that the Google STT + GPT-4.1 + Cartesia TTS stack (C3) achieves the highest means on conversational quality, technical quality, accuracy, and user satisfaction, with Welch ANOVA and Games-Howell post-hoc tests indicating significant differences. The paper also reports weak correlations between LLM-judge metrics and user ratings and proposes a component-level decomposition claiming that STT quality is the dominant factor. The manuscript includes the full evaluation prompt and detailed statistical tables, but several load-bearing limitations are acknowledged in Section 5.1.

Significance. If the causal ranking were established, this would be a valuable large-scale empirical comparison of cascaded speech-language systems in a real deployment, with practical guidance for component selection and a useful dual-evaluation methodology. The paper's strengths include the scale of the production data, the transparent disclosure of the full LLM-judge prompt, and the use of standard statistical procedures with effect sizes. However, the central claims about C3's superiority and the component-level attribution depend on comparisons across non-overlapping deployment periods, a non-factorial design, and an LLM judge whose agreement with human judgments is not demonstrated in this manuscript. The weak objective-user correlation is an interesting and falsifiable finding, but it does not by itself compensate for the causal identification gaps.

major comments (4)

[Section 5.1 (Temporal Confounds); Tables 3-4] The manuscript's own limitation statement acknowledges that non-overlapping deployment periods introduce confounds from seasonal variation, candidate pool changes, and system improvements over time. Because C3's superiority is reported entirely through comparisons across these non-overlapping periods, the observed higher means and significant Welch ANOVA results cannot be attributed specifically to the STT/LLM/TTS choices. The abstract's phrasing that C3 'outperforms alternatives' is stronger than the evidence supports unless the authors provide deployment date ranges, concurrent control measurements, or a sensitivity analysis showing that period-level factors cannot explain the effect. This is the load-bearing gap for the paper's headline claim.
[Section 4.2 and Abstract (User Satisfaction Claim)] The post-hoc description states that for rate star, C3 was significantly better than C2, C4, and C5, but not C1. The abstract nonetheless claims that C3 outperforms alternatives in user satisfaction scores. Since the pairwise comparison against C1 is not significant, the user-satisfaction component of the central claim is not fully supported. The paper should either report the full post-hoc matrix for rate star, including the C3 vs. C1 comparison, and temper the abstract, or provide additional evidence that the lack of significance is due to sample size or other identifiable factors.
[Section 3.4 and Appendix A (LLM-as-a-Judge Validation)] RQ3 asks whether LLM-based evaluation can reliably assess voice-based AI interactions, and the abstract claims a 'validated evaluation methodology.' The only validation support is a citation to the authors' prior work [1]; this manuscript reports no agreement statistics between the Claude 3.5 Sonnet judge and human raters, no inter-rater reliability, and no calibration analysis. The prompt itself contains specific instructions (e.g., replacing 'Mid-level' with 'Experienced'; scoring only the interviewer's responses, not the candidate's), which may introduce systematic biases that are not evaluated. Without judge-validation evidence with confidence intervals or agreement coefficients, the construct validity of the primary outcome metrics remains unestablished.
[Section 4.5 and Table 1 (Component Contribution Analysis)] The component contribution analysis claims that STT has the dominant impact and that TTS contributes smaller but meaningful improvements. However, the five configurations in Table 1 form a non-factorial design: Google STT appears only with GPT-4.1 or GPT-4o, while Whisper appears only with GPT-4o or Groq2, and Cartesia TTS appears in only one configuration. The reported 'components effects' are therefore confounded with the specific combinations in which each component appears. The statement that Google STT 'consistently outperforms' Whisper STT is not supported by matched comparisons that hold the other two components fixed. A factorial or matched-pair design, or a clear regression model with interaction terms, is needed before these component-level conclusions can be drawn.

minor comments (5)

[Abstract and Section 3.3] The abstract says 'over 300,000 AI-conducted job interviews' while Section 3.3 describes 'over 5,000 AI-conducted interviews, sampled and segmented.' Please clarify whether 300,000 refers to the total population and 5,000 to the analyzed sample, and state the sampling procedure.
[Section 3.1] There are typographical errors: 'Speech-to-T ext' and 'T ext-to-Speech' should be 'Speech-to-Text' and 'Text-to-Speech.'
[Table 5] The soft skills row reports both a Welch p-value of 1.000 and a Kruskal-Wallis result with p=0.136; the table's 'Test Used' column says Kruskal-Wallis, but the text in Section 4.2 uses the Welch p-value. Please specify which test was used for the soft skills metric and clarify the discrepancy.
[Section 4.5] The sentence 'The analysis tells important findings about component contributions' appears to be missing a verb; it should likely read 'reveals important findings' or 'provides important findings.'
[Section 4.4 and Figures 1-2] The figures are referenced in the text but the figure captions do not include the numerical values or confidence intervals. Adding error bars or boxplot annotations would help readers assess the overlap in distributions, especially given the small effect sizes.

Circularity Check

1 steps flagged · score 4.0 of 10

One self-citation carries the 'validated evaluation framework' claim, but the C3 ranking itself rests on independent score comparisons.

self citation load bearing [Section 3.4 (Evaluation Framework) and Reference [1]]
"We employ a dual evaluation approach, combining direct user feedback with an automated, LLM-based assessment framework validated against human data [1]."

The only cited support for the claim that the LLM-as-a-Judge framework is 'validated against human data' is reference [1], whose author list overlaps with the present paper (Ali Ansari and Nima Yazdani). No external validation, reproduced human-correlation analysis, or independent benchmark is supplied in this paper; moreover, Section 4.6 reports weak correlations between the judge metrics and user ratings, so the current data do not independently establish the judge's validity. The paper's 'validated evaluation methodology' contribution and the affirmative answer to RQ3 therefore rest on a self-citation rather than on independent evidence.

full rationale

The main empirical comparison of STT x LLM x TTS stacks is not circular: the LLM judge is a separate model (Claude 3.5 Sonnet) from the LLMs being benchmarked, the five configurations are independently defined in Table 1, and the reported means, Welch ANOVA results, and Games-Howell post-hoc comparisons are computed directly from production data. The user-satisfaction ratings come from candidate feedback, not from the judge, so the headline ranking is not definitionally tied to its inputs. The only circularity-adjacent element is the 'validated against human data [1]' citation in Section 3.4: reference [1] shares two authors with the present paper, and no external validation is shown here, so the 'validated evaluation methodology' contribution relies on a self-citation. However, this does not force the main C3-superiority result, which would stand as a descriptive comparison even if the judge's validity were set aside. Threats such as non-overlapping deployment periods or self-selection bias are validity concerns, not circular reasoning, and therefore do not raise the circularity score further.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted free parameters appear in the paper; the reported scores are measurements from an LLM judge and user stars. The principal assumptions are that the judge measures quality, that the sampled interviews are representative, that non-overlapping deployment periods are not confounded by time, and that component effects can be read from five non-factorial configurations.

assumptions (4)

ad hoc to paper LLM-as-a-Judge scores are valid proxies for conversational and technical quality.
The paper uses Claude 3.5 Sonnet scores as objective metrics and says the framework was validated against human data [1], but includes no in-paper validation or agreement statistics. Invoked in Section 3.4 and Appendix A.
domain assumption The five configurations were deployed in periods without unobserved temporal confounds.
Needed to attribute differences to components rather than to timing or candidate pool changes. The paper acknowledges this assumption is doubtful in Section 5.1 Temporal Confounds.
domain assumption The sampled 5,000 interviews are representative of the production interview population.
The sampling scheme and inclusion criteria are not specified in Section 3.3, and Section 5.1 notes that user ratings may be skewed by self-selection.
ad hoc to paper Component effects can be inferred from the five non-factorial configurations.
Section 4.5 infers that STT has the dominant impact from comparisons such as C3 versus C5, which change STT, LLM, and TTS simultaneously, so the component attribution is an inferential assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems." pith.science (2026). https://pith.science/paper/FUBBGDNQ

@misc{pith2026250716835,
  author       = {Pith},
  title        = {Pith review of: Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUBBGDNQ}},
  note         = {Machine review of arXiv:2507.16835}
}

read the original abstract

Voice-based conversational AI systems increasingly rely on cascaded architectures that combine speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS) components. We present a large-scale empirical comparison of STT x LLM x TTS stacks using data sampled from over 300,000 AI-conducted job interviews. We used an LLM-as-a-Judge automated evaluation framework to assess conversational quality, technical accuracy, and skill assessment capabilities. Our analysis of five production configurations reveals that a stack combining Google's STT, GPT-4.1, and Cartesia's TTS outperforms alternatives in both objective quality metrics and user satisfaction scores. Surprisingly, we find that objective quality metrics correlate weakly with user satisfaction scores, suggesting that user experience in voice-based AI systems depends on factors beyond technical performance. Our findings provide practical guidance for selecting components in multimodal conversations and contribute a validated evaluation methodology for human-AI interactions.

Figures

Figures reproduced from arXiv: 2507.16835 by the authors.

**Figure 2.** Combined boxplot showing Conversational Quality (top), Technical Quality (bottom left), and User [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Component Contribution Analysis showing individual STT, LLM, and TTS effects across key [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Correlation matrix showing relationships between LLM-as-a-Judge metrics and user satisfaction. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages

[1]

Better together: Quantifying the benefits of ai-assisted recruitment, 2025

Ada Aka, Emil Palikot, Ali Ansari, and Nima Yazdani. Better together: Quantifying the benefits of ai-assisted recruitment, 2025

work page 2025
[2]

ESPnet-SDS: A unified all-in-one speech-to-dialogue system

Ashish Arora, Shinji Watanabe, and Takaaki Hori. ESPnet-SDS: A unified all-in-one speech-to-dialogue system. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025

work page 2025
[3]

Machine learning and information theory concepts towards an AI Mathematician

Antoine Ginart, Aruj Mahajan, and Ali Ansari. Asynchronous agent-based architecture for real-time voice-based conversational agents. arXiv preprint arXiv:2403.04571, 2024

work page Pith review arXiv 2024
[4]

Audiogpt: Understanding and generating speech, music, sound, and talking head

Rongjie Huang, Jiawei Chen, Ziyang Wang, Zhuo Liu, Yuan Gong, Chen Li, Yang Zhang, Yi Liu, Jing Liu, Lewen Li, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5550–5561, 2023. 10

work page 2023
[5]

Lslm: A listening while speaking language model for real-time full-duplex dialogue

Zhaofeng Ma, Zhaoyuan Lin, Yuxiang Wang, Zhen Bi, Yidong Peng, Shiyin Jin, and Zhen Liu. Lslm: A listening while speaking language model for real-time full-duplex dialogue. arXiv preprint arXiv:2405.02324, 2024

work page arXiv 2024
[6]

The Dirac equation on metrics of Eguchi-Hanson type

Dong Zhang, Shimin Li, Wenhua Gan, Ziteng Wang, Jian Wu, Quan Liu, and Long Chen. Speechgpt: A large language model for speech. arXiv preprint arXiv:2305.18344, 2023

work page Pith review arXiv 2023
[7]

{index}" - LLM model: {model} - interview_transcript:

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In International Conference on Learning Representations, 2024. 11 A LLM-as-a-Judge System Prompt The following prompt was used with the ‘claude-3-5-sonnet-20241022‘...

work page 2024
[8]

**Score the Evaluations**: - **Accuracy**: Does the evaluation align with the transcript and correctly reflect the candidate’s performance?

work page

Show all 9 references

[9]

Mid-level

**Evaluate Interview Quality**: - Score both conversational and technical aspects of the interview --- ### Ensure Ratings Are Standardized** - **Replace all occurrences of "Mid-level" (case-insensitive) with "Experienced"** in ‘evaluation_result‘. - Ensure that all **evaluatio...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Better together: Quantifying the benefits of ai-assisted recruitment, 2025

Ada Aka, Emil Palikot, Ali Ansari, and Nima Yazdani. Better together: Quantifying the benefits of ai-assisted recruitment, 2025

work page 2025

[2] [2]

ESPnet-SDS: A unified all-in-one speech-to-dialogue system

Ashish Arora, Shinji Watanabe, and Takaaki Hori. ESPnet-SDS: A unified all-in-one speech-to-dialogue system. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025

work page 2025

[3] [3]

Machine learning and information theory concepts towards an AI Mathematician

Antoine Ginart, Aruj Mahajan, and Ali Ansari. Asynchronous agent-based architecture for real-time voice-based conversational agents. arXiv preprint arXiv:2403.04571, 2024

work page Pith review arXiv 2024

[4] [4]

Audiogpt: Understanding and generating speech, music, sound, and talking head

Rongjie Huang, Jiawei Chen, Ziyang Wang, Zhuo Liu, Yuan Gong, Chen Li, Yang Zhang, Yi Liu, Jing Liu, Lewen Li, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5550–5561, 2023. 10

work page 2023

[5] [5]

Lslm: A listening while speaking language model for real-time full-duplex dialogue

Zhaofeng Ma, Zhaoyuan Lin, Yuxiang Wang, Zhen Bi, Yidong Peng, Shiyin Jin, and Zhen Liu. Lslm: A listening while speaking language model for real-time full-duplex dialogue. arXiv preprint arXiv:2405.02324, 2024

work page arXiv 2024

[6] [6]

The Dirac equation on metrics of Eguchi-Hanson type

Dong Zhang, Shimin Li, Wenhua Gan, Ziteng Wang, Jian Wu, Quan Liu, and Long Chen. Speechgpt: A large language model for speech. arXiv preprint arXiv:2305.18344, 2023

work page Pith review arXiv 2023

[7] [7]

{index}" - LLM model: {model} - interview_transcript:

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In International Conference on Learning Representations, 2024. 11 A LLM-as-a-Judge System Prompt The following prompt was used with the ‘claude-3-5-sonnet-20241022‘...

work page 2024

[8] [8]

**Score the Evaluations**: - **Accuracy**: Does the evaluation align with the transcript and correctly reflect the candidate’s performance?

work page